View cluster autoscaler events

This page provides information about visibility events emitted by cluster autoscaler in Google Kubernetes Engine (GKE). By analyzing these events, you can gain insights into how cluster autoscaler manages your cluster's scaling and understand the reasons behind its decisions.

The GKE cluster autoscaler emits visibility events, which are available as log entries in Cloud Logging. The events described in this guide are separate from the Kubernetes events produced by the cluster autoscaler.

Requirements

To see autoscaler events, you must enable Cloud Logging in your cluster. The events won't be produced if Logging is disabled.

Viewing events

The visibility events for the cluster autoscaler are stored in a Cloud Logging log, in the same project as where your GKE cluster is located. You can also view these events from the notifications in the Google Kubernetes Engine page in Google Cloud console.

Viewing visibility event logs

To view the logs, perform the following:

1. In the Google Cloud console, go to the Kubernetes Clusters page.

   Go to Kubernetes Clusters

2. Select the name of your cluster to view its Cluster Details page.

3. On the Cluster Details page, click the Logs tab.

4. On the Logs tab, click the Autoscaler Logs tab to view the logs.

5. (Optional) To apply more advanced filters to narrow the results, click the button with the arrow on the right side of the page to view the logs in Logs Explorer.

Viewing visibility event notifications

To view the visibility event notifications on the Google Kubernetes Engine page, perform the following:

1. Go to the Google Kubernetes Engine page in the Google Cloud console:

   Go to Google Kubernetes Engine

2. Check the Notifications column for specific clusters to find notifications related to scaling.

3. Click the notification for detailed information, recommended actions, and to access the logs for this event.

Types of events

All logged events are in the JSON format and can be found in the jsonPayload field of a log entry. All timestamps in the events are UNIX second timestamps.

Here's a summary of the types of events emitted by the cluster autoscaler:

Status event

A status event is emitted periodically, and describes the actual size of all autoscaled node pools and the target size of all autoscaled node pools as observed by cluster autoscaler.

Example

The following log sample shows a status event:

{
  "status": {
    "autoscaledNodesCount": 4,
    "autoscaledNodesTarget": 4,
    "measureTime": "1582898536"
  }
}

ScaleUp event

A scaleUp event is emitted when the cluster autoscaler scales the cluster up. The autoscaler increases the size of the cluster's node pools by scaling up the underlying Managed Instance Groups (MIGs) for the node pools. To learn more about how scale up works, see How does scale up work? in the Kubernetes Cluster Autoscaler FAQ.

The event contains information on which MIGs were scaled up, by how many nodes, and which unschedulable Pods triggered the event.

The list of triggering Pods is truncated to 50 arbitrary entries. The actual number of triggering Pods can be found in the triggeringPodsTotalCount field.

Example

The following log sample shows a scaleUp event:

{
  "decision": {
    "decideTime": "1582124907",
    "eventId": "ed5cb16d-b06f-457c-a46d-f75dcca1f1ee",
    "scaleUp": {
      "increasedMigs": [
        {
          "mig": {
            "name": "test-cluster-default-pool-a0c72690-grp",
            "nodepool": "default-pool",
            "zone": "us-central1-c"
          },
          "requestedNodes": 1
        }
      ],
      "triggeringPods": [
        {
          "controller": {
            "apiVersion": "apps/v1",
            "kind": "ReplicaSet",
            "name": "test-85958b848b"
          },
          "name": "test-85958b848b-ptc7n",
          "namespace": "default"
        }
      ],
      "triggeringPodsTotalCount": 1
    }
  }
}

ScaleDown event

A scaleDown event is emitted when cluster autoscaler scales the cluster down. To learn more about how scale down works, see How does scale down work? in the Kubernetes Cluster Autoscaler FAQ.

The cpuRatio and memRatio fields describe the CPU and memory utilization of the node, as a percentage. This utilization is a sum of Pod requests divided by node allocatable, not real utilization.

The list of evicted Pods is truncated to 50 arbitrary entries. The actual number of evicted Pods can be found in the evictedPodsTotalCount field.

Use the following query to verify if the cluster autoscaler scaled down the nodes:

resource.type="k8s_cluster" \
resource.labels.location=COMPUTE_REGION \
resource.labels.cluster_name=CLUSTER_NAME \
log_id("container.googleapis.com/cluster-autoscaler-visibility") \
( "decision" NOT "noDecisionStatus" )

Replace the following:

• CLUSTER_NAME: the name of the cluster.

• COMPUTE_REGION: the cluster's Compute Engine region, such as us-central1.

Example

The following log sample shows a scaleDown event:

{
  "decision": {
    "decideTime": "1580594665",
    "eventId": "340dac18-8152-46ff-b79a-747f70854c81",
    "scaleDown": {
      "nodesToBeRemoved": [
        {
          "evictedPods": [
            {
              "controller": {
                "apiVersion": "apps/v1",
                "kind": "ReplicaSet",
                "name": "kube-dns-5c44c7b6b6"
              },
              "name": "kube-dns-5c44c7b6b6-xvpbk"
            }
          ],
          "evictedPodsTotalCount": 1,
          "node": {
            "cpuRatio": 23,
            "memRatio": 5,
            "mig": {
              "name": "test-cluster-default-pool-c47ef39f-grp",
              "nodepool": "default-pool",
              "zone": "us-central1-f"
            },
            "name": "test-cluster-default-pool-c47ef39f-p395"
          }
        }
      ]
    }
  }
}

You can also view the scale-down event on the nodes with no workload running (typically only system pods created by DaemonSets).

Use the following query to see the event logs:

resource.type="k8s_cluster" \
resource.labels.project_id=PROJECT_ID \
resource.labels.location=COMPUTE_REGION \
resource.labels.cluster_name=CLUSTER_NAME \
severity>=DEFAULT \
logName="projects/PROJECT_ID/logs/events" \
("Scale-down: removing empty node")

Replace the following:

• PROJECT_ID: your project ID.

• CLUSTER_NAME: the name of the cluster.

• COMPUTE_REGION: the cluster's Compute Engine region, such as us-central1.

EventResult event

An eventResult event is emitted when a scaleUp or a scaleDown event completes successfully or unsuccessfully. This event contains a list of event IDs (from the eventId field in scaleUp or scaleDown events), along with error messages. An empty error message indicates the event completed successfully. A list of eventResult events are aggregated in the results field.

To diagnose errors, consult the ScaleUp errors and ScaleDown errors sections.

Example

The following log sample shows an eventResult event:

{
  "resultInfo": {
    "measureTime": "1582878896",
    "results": [
      {
        "eventId": "2fca91cd-7345-47fc-9770-838e05e28b17"
      },
      {
        "errorMsg": {
          "messageId": "scale.down.error.failed.to.delete.node.min.size.reached",
          "parameters": [
            "test-cluster-default-pool-5c90f485-nk80"
          ]
        },
        "eventId": "ea2e964c-49b8-4cd7-8fa9-fefb0827f9a6"
      }
    ]
  }
}

NodePoolCreated event

A nodePoolCreated event is emitted when cluster autoscaler with node auto-provisioning enabled creates a new node pool. This event contains the name of the created node pool and a list of its underlying MIGs. If the node pool was created because of a scaleUp event, the eventId of the corresponding scaleUp event is included in the triggeringScaleUpId field.

Example

The following log sample shows a nodePoolCreated event:

{
  "decision": {
    "decideTime": "1585838544",
    "eventId": "822d272c-f4f3-44cf-9326-9cad79c58718",
    "nodePoolCreated": {
      "nodePools": [
        {
          "migs": [
            {
              "name": "test-cluster-nap-n1-standard--b4fcc348-grp",
              "nodepool": "nap-n1-standard-1-1kwag2qv",
              "zone": "us-central1-f"
            },
            {
              "name": "test-cluster-nap-n1-standard--jfla8215-grp",
              "nodepool": "nap-n1-standard-1-1kwag2qv",
              "zone": "us-central1-c"
            }
          ],
          "name": "nap-n1-standard-1-1kwag2qv"
        }
      ],
      "triggeringScaleUpId": "d25e0e6e-25e3-4755-98eb-49b38e54a728"
    }
  }
}

NodePoolDeleted event

A nodePoolDeleted event is emitted when cluster autoscaler with node auto-provisioning enabled deletes a node pool.

Example

The following log sample shows a nodePoolDeleted event:

{
  "decision": {
    "decideTime": "1585830461",
    "eventId": "68b0d1c7-b684-4542-bc19-f030922fb820",
    "nodePoolDeleted": {
      "nodePoolNames": [
        "nap-n1-highcpu-8-ydj4ewil"
      ]
    }
  }
}

NoScaleUp event

A noScaleUp event is periodically emitted when there are unschedulable Pods in the cluster and cluster autoscaler cannot scale the cluster up to accommodate the Pods.

• noScaleUp events are best-effort, that is, these events don't cover all possible reasons for why cluster autoscaler cannot scale up.
• noScaleUp events are throttled to limit the produced log volume. Each persisting reason is only emitted every couple of minutes.
• All the reasons can be arbitrarily split across multiple events. For example, there is no guarantee that all rejected MIG reasons for a single Pod group will appear in the same event.
• The list of unhandled Pod groups is truncated to 50 arbitrary entries. The actual number of unhandled Pod groups can be found in the unhandledPodGroupsTotalCount field.

Reason fields

The following fields help to explain why scaling up did not occur:

• reason: Provides a global reason for why cluster autoscaler is prevented from scaling up. Refer to the NoScaleUp top-level reasons section for details.
• napFailureReason: Provides a global reason preventing cluster autoscaler from provisioning additional node pools (for example, node auto-provisioning is disabled). Refer to the NoScaleUp top-level node auto-provisioning reasons section for details.
• skippedMigs[].reason: Provides information about why a particular MIG was skipped. Cluster autoscaler skips some MIGs from consideration for any Pod during a scaling up attempt (for example, because adding another node would exceed cluster-wide resource limits). Refer to the NoScaleUp MIG-level reasons section for details.
• unhandledPodGroups: Contains information about why a particular group of unschedulable Pods does not trigger scaling up. The Pods are grouped by their immediate controller. Pods without a controller are in groups by themselves. Each Pod group contains an arbitrary example Pod and the number of Pods in the group, as well as the following reasons:
  • napFailureReasons: Reasons why cluster autoscaler cannot provision a new node pool to accommodate this Pod group (for example, Pods have affinity constraints). Refer to the NoScaleUp Pod-level node auto-provisioning reasons section for details.
  • rejectedMigs[].reason: Per-MIG reasons why cluster autoscaler cannot increase the size of a particular MIG to accommodate this Pod group (for example, the MIG's node is too small for the Pods). Refer to the NoScaleUp MIG-level reasons section for details.

Example

The following log sample shows a noScaleUp event:

{
  "noDecisionStatus": {
    "measureTime": "1582523362",
    "noScaleUp": {
      "skippedMigs": [
        {
          "mig": {
            "name": "test-cluster-nap-n1-highmem-4-fbdca585-grp",
            "nodepool": "nap-n1-highmem-4-1cywzhvf",
            "zone": "us-central1-f"
          },
          "reason": {
            "messageId": "no.scale.up.mig.skipped",
            "parameters": [
              "max cluster cpu limit reached"
            ]
          }
        }
      ],
      "unhandledPodGroups": [
        {
          "napFailureReasons": [
            {
              "messageId": "no.scale.up.nap.pod.zonal.resources.exceeded",
              "parameters": [
                "us-central1-f"
              ]
            }
          ],
          "podGroup": {
            "samplePod": {
              "controller": {
                "apiVersion": "v1",
                "kind": "ReplicationController",
                "name": "memory-reservation2"
              },
              "name": "memory-reservation2-6zg8m",
              "namespace": "autoscaling-1661"
            },
            "totalPodCount": 1
          },
          "rejectedMigs": [
            {
              "mig": {
                "name": "test-cluster-default-pool-b1808ff9-grp",
                "nodepool": "default-pool",
                "zone": "us-central1-f"
              },
              "reason": {
                "messageId": "no.scale.up.mig.failing.predicate",
                "parameters": [
                  "NodeResourcesFit",
                  "Insufficient memory"
                ]
              }
            }
          ]
        }
      ],
      "unhandledPodGroupsTotalCount": 1
    }
  }
}

NoScaleDown event

A noScaleDown event is periodically emitted when there are nodes which are blocked from being deleted by cluster autoscaler.

• Nodes that cannot be removed because their utilization is high are not included in noScaleDown events.
• NoScaleDown events are best effort, that is, these events don't cover all possible reasons for why cluster autoscaler cannot scale down.
• NoScaleDown events are throttled to limit the produced log volume. Each persisting reason will only be emitted every couple of minutes.
• The list of nodes is truncated to 50 arbitrary entries. The actual number of nodes can be found in the nodesTotalCount field.

Reason fields

The following fields help to explain why scaling down did not occur:

• reason: Provides a global reason for why cluster autoscaler is prevented from scaling down (for example, a backoff period after recently scaling up). Refer to the NoScaleDown top-level reasons section for details.
• nodes[].reason: Provides per-node reasons for why cluster autoscaler is prevented from deleting a particular node (for example, there's no place to move the node's Pods to). Refer to the NoScaleDown node-level reasons section for details.

Example

The following log sample shows a noScaleDown event:

{
  "noDecisionStatus": {
    "measureTime": "1582858723",
    "noScaleDown": {
      "nodes": [
        {
          "node": {
            "cpuRatio": 42,
            "mig": {
              "name": "test-cluster-default-pool-f74c1617-grp",
              "nodepool": "default-pool",
              "zone": "us-central1-c"
            },
            "name": "test-cluster-default-pool-f74c1617-fbhk"
          },
          "reason": {
            "messageId": "no.scale.down.node.no.place.to.move.pods"
          }
        }
      ],
      "nodesTotalCount": 1,
      "reason": {
        "messageId": "no.scale.down.in.backoff"
      }
    }
  }
}

Messages

The events emitted by the cluster autoscaler use parameterized messages to provide explanations for the event. The parameters field is available with the messageId field, such as in this example log for a NoScaleUp event.

This section provides descriptions for various messageId and its corresponding parameters. However, this section does not contain all possible messages, and may be extended at any time.

ScaleUp errors

You can find event error messages for scaleUp events in the corresponding eventResult event, in the resultInfo.results[].errorMsg field.

Reasons for a noScaleUp event

A noScaleUp event is periodically emitted when there are unschedulable Pods in the cluster and cluster autoscaler cannot scale the cluster up to schedule the Pods. noScaleUp events are best-effort, and don't cover all possible cases.

NoScaleUp top-level reasons

Top-level reason messages for noScaleUp events appear in the noDecisionStatus.noScaleUp.reason field. The message contains a top-level reason for why cluster autoscaler cannot scale the cluster up.

NoScaleUp top-level node auto-provisioning reasons

Top-level node auto-provisioning reason messages for noScaleUp events appear in the noDecisionStatus.noScaleUp.napFailureReason field. The message contains a top-level reason for why cluster autoscaler cannot provision new node pools.

NoScaleUp MIG-level reasons

MIG-level reason messages for noScaleUp events appear in the noDecisionStatus.noScaleUp.skippedMigs[].reason and noDecisionStatus.noScaleUp.unhandledPodGroups[].rejectedMigs[].reason fields. The message contains a reason why cluster autoscaler can't increase the size of a particular MIG.

NoScaleUp Pod-group-level node auto-provisioning reasons

Pod-group-level node auto-provisioning reason messages for noScaleUp events appear in the noDecisionStatus.noScaleUp.unhandledPodGroups[].napFailureReasons[] field. The message contains a reason why cluster autoscaler cannot provision a new node pool to schedule a particular Pod group.

ScaleDown errors

You can find error event messages for scaleDown events in the corresponding eventResult event, in the resultInfo.results[].errorMsg field.

Reasons for a noScaleDown event

A noScaleDown event is periodically emitted when there are nodes which are blocked from being deleted by cluster autoscaler. noScaleDown events are best-effort, and don't cover all possible cases.

NoScaleDown top-level reasons

Top-level reason messages for noScaleDown events appear in the noDecisionStatus.noScaleDown.reason field. The message contains a top-level reason why cluster autoscaler can't scale the cluster down.

NoScaleDown node-level reasons

Node-level reason messages for noScaleDown events appear in the noDecisionStatus.noScaleDown.nodes[].reason field. The message contains a reason why cluster autoscaler can't remove a particular node.

What's next

• Learn more about cluster autoscaler.
• Learn about how to use node auto-provisioning.
• Troubleshoot cluster autoscaler not scaling down.
• Troubleshoot cluster autoscaler not scaling up.
• Watch a YouTube video about troubleshooting and resolving scaling issues.