Cluster notifications

This page describes how Google Kubernetes Engine (GKE) publishes cluster notifications to Pub/Sub with information about events relevant to your cluster configuration, such as available or ongoing upgrades, and security bulletins.

To learn how to set up cluster notifications, refer to Receive cluster notifications.

Overview

When certain events occur that are relevant to your GKE clusters, such as important available upgrades or security bulletins, GKE publishes notifications about those events as messages to Pub/Sub topics that you configure. You can receive these notifications on a Pub/Sub subscription , integrate with third-party services, and filter for the notification types you want to receive.

Benefits

Using Pub/Sub to receive cluster notifications provides the following benefits:

  • You are notified when security bulletins that are specific to your clusters are issued, which provides you with accurate risk and impact information.
  • You are notified when there is a new GKE version available for your cluster, allowing you to better plan for testing and qualifications, and to help ensure a smooth and predictable upgrade process. Previously, you had to check the GKE release notes or the GKE API to discover when a new GKE version was released.
  • You are notified when either GKE or a user initiates cluster upgrades, and when the upgrade operation finishes, providing you with more visibility into the background operations of your cluster.
  • Pub/Sub is highly extensible, giving you flexibility in how you process incoming notifications. For example, you could integrate with Slack to forward notifications to a Slack channel, or initiate Cloud Run functions to run custom processes.
  • When custom processes are required (for example, orchestrating a staging to production workflow to test and certify an upgrade), you can use the notification to auto-trigger these workflows.

Types of upgrade notifications

GKE sends the following cluster notification types:

You can filter the notifications you receive so that you're only notified for relevant events. You can filter cluster notifications by specifying a value for filter in the --notification-config flag when you enable cluster notifications, or by configuring your Pub/Sub subscription.

SecurityBulletinEvent

When GKE issues a security bulletin that directly correlates to your cluster configuration or version, GKE sends a SecurityBulletinEvent notification, providing you with information about the vulnerability, the impact, and, if applicable, actions you can take.

UpgradeAvailableEvent

When a new version becomes available on a release channel, GKE sends an UpgradeAvailableEvent notification to clusters on that release channel to inform the clusters that a new version is now available. This notification provides one week of advance notice for patch versions and at least 2-4 weeks for minor versions (depending on the channel). For more information, refer to What versions are available in a channel.

For clusters not on a release channel, GKE sends UpgradeAvailableEvent notifications for all new versions that the clusters can upgrade to, including patches on the current minor version and the next minor version.

If you use Windows Server node pools, Windows version information is sent as part of the UpgradeAvailableEvent notification.

UpgradeEvent

When you or GKE initiates an upgrade, GKE sends an UpgradeEvent notification, telling you that an upgrade has begun. Ideally, you should use the UpgradeAvailableEvent notification type to be aware of the upcoming upgrade so that you can either upgrade in advance or take necessary measures to prepare, such as setting up maintenance windows.

The UpgradeEvent notification is sent at the start of the upgrade operation. The operation ID is passed in the message.

UpgradeInfoEvent

When GKE finishes the operation to automatically or manually upgrade a cluster's control plane or nodes, GKE sends an UpgradeInfoEvent notification to inform you that the operation is complete. The operation completes with one of the following states:

  • SUCCEEDED: GKE successfully upgraded the control plane or nodes.
  • FAILED: GKE failed to upgrade the control plane or nodes.
  • CANCELED: GKE canceled the upgrade operation for technical or business reasons, or you canceled the upgrade operation.

Use the notification to confirm the success of an upgrade operation.

Filtering notifications

You can filter cluster notifications to ensure that you receive only the notifications that you want. You can apply filtering in one of the following ways:

Filtering notifications in GKE

You can set up filtering for one or more available notification types when enabling cluster notifications by specifying values for filter in the --notification-config flag. filter takes a pipe ( | ) delimited list of the notification types you want to receive.

For example, specifying filter="UpgradeEvent|SecurityBulletinEvent" tells GKE to only send notifications for UpgradeEvent and SecurityBulletinEvent notification types.

Filtering notifications using filter has benefits such as the following:

  • Easier to use, because you filter on the notification type without using a specific syntax.
  • Notifications you filter out are never sent to Pub/Sub, so you aren't charged fees for undelivered messages.
  • You can edit the filter configuration at any time.

For instructions on filtering notifications in GKE, see Receive cluster notifications.

Filtering notifications in Pub/Sub

Pub/Sub supports filtering messages in your subscription using a filtering syntax. When you use this method, GKE delivers all notification types to your Pub/Sub topic. Pub/Sub filters messages based on your subscription configuration and delivers the messages you want to receive.

For example, you could filter for UpgradeEvent and UpgradeAvailableEvent notifications using the following syntax in your subscription:

attributes.type_url = "type.googleapis.com/google.container.v1beta1.UpgradeEvent" OR "type.googleapis.com/google.container.v1beta1.UpgradeAvailableEvent"

You are still charged for undelivered messages filtered by your subscription. You also cannot modify the filters after you've configured the subscription. However, the filtering syntax is more extensible than filtering in GKE.

To learn more about filtering your Pub/Sub subscription, see Filtering messages.

Consuming Pub/Sub messages

Pub/Sub messages contain two fields: data (string) and attributes (string-to-string map).

For GKE notifications, the data field contains human-readable information. The attributes field has generic notification information like the notification type, your project ID, cluster name, and cluster location. The attributes.payload field is a JSON-parsable string that contains specific notification information, such as the details of a security bulletin.

Notifications always contain the following attributes:

Attribute Description Example
project_id The project number that owns the cluster. 123456789
cluster_location The location of the cluster. us-central1-c
cluster_name The name of the cluster. example-cluster
type_url The type of notification. type.googleapis.com/google.container.v1beta1.UpgradeEvent
payload A JSON-parsable string carrying notification-specific information. 
{ "resourceType":"MASTER",
  "operation":"operation-1595889094437-87b7254a",
  "operationStartTime":"2020-07-27T22:31:34.437652293Z",
  "currentVersion":"1.15.12-gke.2",
  "targetVersion":"1.15.12-gke.9"}

GKE will always send beta notification types. However, you can parse the payload to display the corresponding GA notification type if it is available.

Sample cluster notification messages

In addition to the text in the data field, each message that GKE sends to Pub/Sub has specific values in the attributes.type_url and attributes.payload fields. The following tables show you examples of the information you might receive for each notification type:

SecurityBulletinEvent

The output is similar to the following for a SecurityBulletinEvent message:

Attributes
type_url type.googleapis.com/google.container.v1beta1.SecurityBulletinEvent
payload 
{    "resourceTypeAffected":"RESOURCE_TYPE_CONTROLPLANE",
         "bulletinId":"GCP-2021-001",
         "cveIds":[
            "CVE-2021-3156"
         ],
         "severity":"Medium",
         "briefDescription":"A vulnerability was recently discovered in the Linux utility sudo, described in CVE-2021-3156, that may allow an attacker with unprivileged local shell access on a system with sudo installed to escalate their privileges to root on the system.",
         "affectedSupportedMinors":["1.18", "1.19"],
         "patchedVersions":["1.18.9-gke.1900", "1.19.9-gke.1900"],
         "suggestedUpgradeTarget":"1.19.9-gke.1900",
         "bulletinUri":"https://cloud.google.com/anthos/clusters/docs/security-bulletins#gcp-2021-001"
      }

UpgradeAvailableEvent

The output is similar to the following for an UpgradeAvailableEvent message:

Attributes
type_url type.googleapis.com/google.container.v1beta1.UpgradeAvailableEvent
payload 
{ "version":"1.17.15-gke.800",
  "resourceType":"MASTER",
  "releaseChannel":{"channel":"RAPID"},
  "windowsVersions": [
    {
      "imageType": "WINDOWS_SAC",
      "osVersion": "10.0.18363.1198",
      "supportEndDate": {
        "day": 10,
        "month": 5,
        "year": 2022
      }
    },
    {
      "imageType": "WINDOWS_LTSC",
      "osVersion": "10.0.17763.1577",
      "supportEndDate": {
        "day": 9,
        "month": 1,
        "year": 2024
      }
    }
  ]
}

UpgradeEvent

The output is similar to the following for an UpgradeEvent message:

Attributes
type_url type.googleapis.com/google.container.v1beta1.UpgradeEvent
payload 
{ "resourceType":"MASTER",
  "operation":"operation-1595889094437-87b7254a",
  "operationStartTime":"2020-07-27T22:31:34.437652293Z",
  "currentVersion":"1.15.12-gke.2",
  "targetVersion":"1.15.12-gke.9"}

UpgradeInfoEvent

The output is similar to the following for an UpgradeInfoEvent message, in this example for a node pool upgrade:

Attributes
type_url type.googleapis.com/google.container.v1beta1.UpgradeInfoEvent
payload 
{ "currentVersion":"1.31.1-gke.1846000",
  "endTime":"2024-11-06T17:12:54.111640650Z",
  "operation":"operation-1730912205658-de2f88a8-6290-4718-b2c1-fb19611060b8",
  "resource":"projects/PROJECT_ID/locations/CLUSTER_LOCATION/clusters/CLUSTER_NAME/nodePools/NODE_POOL_NAME",
  "resourceType":"NODE_POOL"
  "startTime":"2024-11-06T16:56:45.658321844Z",
  "state":"SUCCEEDED",
  "targetVersion":"1.31.1-gke.2105000"}

What's next