Manage host events across reservations

This document explains how to monitor, plan for, and perform pending maintenance on your reserved blocks of capacity in Hypercompute Cluster. To manage maintenance on your virtual machine (VM) instances, see instead Manage host events across VMs.

You can proactively manage upcoming maintenance host events on your reserved blocks of capacity, whether VMs are running on them or not. This approach helps you minimize disruptions and maintain optimal performance.

Before you begin

Select the tab for how you plan to use the samples on this page:

Console

When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

gcloud

In the Google Cloud console, activate Cloud Shell.

Activate Cloud Shell

At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

REST

To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

    After installing the Google Cloud CLI, initialize it by running the following command:

    gcloud init

    If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

For more information, see Authenticate for using REST in the Google Cloud authentication documentation.

Required roles

To get the permissions that you need to manage host maintenance events across reservations, ask your administrator to grant you the following IAM roles:

  • Compute Admin (roles/compute.admin) on the project
  • For read-only access to System Event audit logs: Logs Viewer (roles/logging.viewer) on the project

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to manage host maintenance events across reservations. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to manage host maintenance events across reservations:

  • To start host maintenance for a reservation: compute.reservations.performMaintenance on the project
  • To start host maintenance for a block within a reservation: compute.reservationBlocks.performMaintenance on the project
  • To view the details of a reservation: compute.reservations.get on the project

You might also be able to get these permissions with custom roles or other predefined roles.

Overview

To optimize the maintenance of your reserved blocks of capacity, complete the following steps:

  1. Set up notification alerts. Create log-based alerts to get notified about scheduled, started, or completed maintenance events for a reservation or a reservation block. This approach helps you proactively plan your activities and avoid unexpected downtime.

  2. Manage maintenance across blocks of capacity. View and, if needed, manually start maintenance across your reservations or reservation blocks. This process helps you increase the resilience of your workloads to host errors, prevent downtime, and ensure that your applications remain available.

For more information about the frequency and maintenance behavior of your reserved machine types, see Understand host maintenance.

Set up notification alerts for reservations

You can get notified about scheduled, started, or completed maintenance events for a reservation or reservation block by creating log-based alerting policies.

To create an alert for the maintenance events of a reservation or a reservation block, complete the following procedure. Repeat this procedure for each alert that you want to create.

  1. In the Google Cloud console, go to the Logs Explorer page:

    Go to Logs Explorer

    If you use the search bar to find this page, then select the result whose subheading is Logging.

  2. Click the Show query toggle to the on position.

  3. In the Query pane, build one of the following queries. These queries filter log entries to identify specific maintenance events. Repeat this procedure for each query you want to create.

    • Receive alerts for maintenance of a reservation:

      • To receive alerts when maintenance is scheduled:

        protoPayload.methodName="compute.reservations.upcomingGroupMaintenance" severity>=DEFAULT
        protoPayload.status.message =~ "scheduled"
        
      • To receive alerts when maintenance has completed:

        protoPayload.methodName="compute.reservations.completedGroupMaintenance" severity>=DEFAULT
        protoPayload.status.message =~ "completed"
        
    • Receive alerts for maintenance of a block within a reservation:

      • To receive alerts when maintenance is scheduled:

        protoPayload.methodName="compute.reservations.block.upcomingGroupMaintenance" severity>=DEFAULT
        protoPayload.status.message =~ "scheduled"
        
      • To receive alerts when maintenance has started:

        protoPayload.methodName="compute.reservations.block.startGroupMaintenance" severity>=DEFAULT
        protoPayload.status.message =~ "started"
        
      • To receive alerts when maintenance has completed:

        protoPayload.methodName="compute.reservations.block.completedGroupMaintenance" severity>=DEFAULT
        protoPayload.status.message =~ "completed"
        
  4. To validate the query, click Run query. If the query is valid, then the Query results pane displays log entries that match the query.

  5. In the Query results toolbar, click the Actions list, and then select Create log alert. The Create logs-based alert policy pane appears.

  6. In the Alert details section, do the following:

    1. In the Alert Policy Name field, enter a name for the policy.

    2. In the Policy severity level list, select Warning (or a higher severity).

    3. Click Next.

  7. In the Choose logs to include in the alert section, click Next.

  8. In the Set notification frequency and autoclose duration section, specify the following:

    1. In the Time between notifications list, select how often you want to be notified.

    2. In the Incident autoclose duration list, select after how long Cloud Logging stops sending notifications and automatically closes the incident.

    3. Click Next.

  9. In the Who should be notified? section, specify a notification channel for Logging to send notifications to.

  10. Click Save.

Manage maintenance across reservations

You can view and control maintenance for your reservations or reservation blocks by doing one or more of the following:

  • To check the state and scheduled time of upcoming maintenance for your reservations or reservation blocks, view maintenance state.

  • To immediately start maintenance on a reservation or reservation block, rather than waiting for their scheduled maintenance time, manually start maintenance.

View maintenance state

You can view the upcoming maintenance state for a reservation and a reservation block by checking the value of the upcomingGroupMaintenance field in their metadata. If a reservation lacks the upcomingGroupMaintenance field, then no host events are scheduled for the reservation or its blocks. For more information about the fields in upcomingGroupMaintenance, see Maintenance status definitions in the Compute Engine documentation.

You can view the maintenance state for multiple reservations simultaneously or single reservations. For multiple reservations, use the Google Cloud console or REST API. For single reservations, use any of the following options:

Console

  1. In the Google Cloud console, go to the Reservations page.

    Go to Reservations

  2. In the Maintenance status column, Compute Engine displays the maintenance state of your reservations. If you don't see this column in the On-demand reservations table, then click Column display options, select the Maintenance status checkbox, and then click OK.

  3. To view the maintenance state of a reservation block, complete the following steps:

    1. In the Name column, click the name of the reservation. A page that gives the details of the reservation appears.

    2. In the Blocks table, in the Maintenance column, Compute Engine displays the maintenance state of a reservation blocks.

gcloud

  1. To view the maintenance state of a reservation, use the gcloud beta compute reservations describe command with the --flatten flag set to reservationMaintenance:

    gcloud beta compute reservations describe RESERVATION_NAME \
        --flatten=reservationMaintenance \
        --zone=ZONE
    

    Replace the following:

    • RESERVATION_NAME: the name of the reservation.

    • ZONE: the zone where the reservation exists.

    If maintenance is scheduled for your reservation, then the output is similar to the following:

    ---
    maintenanceOngoingCount: 0
    maintenancePendingCount: 4
    schedulingType: GROUPED
    upcomingGroupMaintenance:
      canReschedule: true
      maintenanceStatus: PENDING
      type: SCHEDULED
    

    If maintenance isn't scheduled, then the output is similar to the following:

    ---
    schedulingType: GROUPED
    
  2. If maintenance is scheduled for your reservation, then you can view the maintenance state of the reservation blocks by using the gcloud beta compute reservations blocks list command:

    gcloud beta compute reservations blocks list RESERVATION_NAME \
        --zone=ZONE
    

    If maintenance is scheduled or ongoing for a block, then the block contains the upcomingGroupMaintenance field:

    ---
    name: example-fr-a3u-dense-1-block-0001
    ...
    reservationMaintenance:
      maintenanceOngoingCount: 0
      maintenancePendingCount: 2
      schedulingType: GROUPED
      upcomingGroupMaintenance:
        canReschedule: true
        maintenanceStatus: PENDING
        type: SCHEDULED
        windowEndTime: '2025-05-13T14:00:00.000-08:00'
        windowStartTime: '2025-05-13T12:00:00.000-08:00'
    ...
    ---
    name: example-fr-a3u-dense-1-block-0002
    ...
    reservationMaintenance:
      schedulingType: GROUPED
    ...
    

    If the schedulingType field is set to INDEPENDENT for a block, then the upcomingGroupMaintenance field doesn't contain the windowStartTime and windowEndTime fields. To see when maintenance is scheduled for a VM running on a block, view the maintenance state for the VM.

REST

  1. To view the maintenance state of your reservations, make a GET request to the beta reservations.list method. In the request URL, include the fields and filter query parameters:

    GET https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/zones/ZONE/reservations?fields=items.name,items.machineType,items.reservationMaintenance:&filter=machineType%20eq%20%2E%2AMACHINE_TYPE
    

    Replace the following:

    • PROJECT_ID: the ID of the project where the reservations exist.

    • ZONE: the zone where the reservations exist.

    • MACHINE_TYPE: the reserved machine type in your reservations.

    If maintenance is scheduled for a reservation, then the reservation contains the upcomingGroupMaintenance field, such as in the following output:

    {
      "items": [
        {
          "name": "reservation-01",
          "machineType": "https://www.googleapis.com/compute/beta/projects/example-project/zones/europe-west1-b/machineTypes/MACHINE_TYPE",
          "reservationMaintenance": {
            "maintenanceOngoingCount": 0,
            "maintenancePendingCount": 4,
            "schedulingType": "GROUPED",
            "upcomingGroupMaintenance": {
              "canReschedule": true,
              "maintenanceStatus": "PENDING"
              "type": "SCHEDULED"
            }
          }
        },
        {
          "name": "reservation-02",
          "machineType": "https://www.googleapis.com/compute/beta/projects/example-project/zones/europe-west1-b/machineTypes/MACHINE_TYPE",
          "reservationMaintenance": {
            "schedulingType": "GROUPED"
          }
        }
      ]
    }
    

    Optionally, to further narrow down a list of VMs, set the filter query parameter to a different filter expression.

  2. If maintenance is scheduled for your reservation, then you can view the maintenance state of the reservation blocks by making a GET request to the beta reservationBlocks.get method. In the request URL, include the fields query parameter:

    GET https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/zones/ZONE/reservations/RESERVATION_NAME/reservationBlocks?fields=items.name,items.reservationMaintenance
    

    Replace RESERVATION_NAME with the name of the reservation.

    If maintenance is scheduled for a block, then the block contains the upcomingGroupMaintenance field, such as in the following output:

    {
      "items": [
        {
          "name": "example-fr-a3u-dense-1-block-0001",
          "reservationMaintenance": {
            "maintenanceOngoingCount": 0,
            "maintenancePendingCount": 2,
            "schedulingType": "GROUPED",
            "upcomingGroupMaintenance": {
              "canReschedule": true,
              "maintenanceStatus": "PENDING",
              "type": "SCHEDULED",
              "windowStartTime": "2025-05-13T14:00:00.000-08:00",
              windowEndTime: "2025-05-13T12:00:00.000-08:00"
            }
          }
        },
        {
          "name": "example-fr-a3u-dense-1-block-0002",
          "reservationMaintenance": {
            "schedulingType": "GROUPED"
          },
        }
      ]
    }
    

    If the schedulingType field is set to INDEPENDENT for a block, then the upcomingGroupMaintenance field doesn't contain the windowStartTime and windowEndTime fields. To see when maintenance is scheduled for a VM running on a block, view the maintenance state for the VM.

Manually start maintenance

You can manually start maintenance for your reservations or reservation blocks instead of waiting for the scheduled time.

Depending on the maintenance state of a reservation or reservation block, the following occurs:

Maintenance state Description What you see
Scheduled Compute Engine has scheduled maintenance for the reservation. You can manually start maintenance before the scheduled time.
  • In the Google Cloud console, the maintenance state shows as Ready to run - will run on DATE.
  • In the gcloud CLI or REST API, Compute Engine sets the maintenanceStatus field to PENDING.
In progress Maintenance is underway. You can't reschedule it.
  • In the Google Cloud console, the maintenance state shows as Running.
  • In the gcloud CLI or REST API, Compute Engine sets the maintenanceStatus field to ONGOING.
Complete Maintenance is finished. Compute Engine has removed all maintenance notifications from the VM.
  • In the Google Cloud console, the maintenance state shows as .
  • In the gcloud CLI or REST API, the maintenanceStatus field doesn't exist.

To manually start maintenance for a reservation or a reservation block, select one of the following options. To start maintenance on specific hosts within a reservation block, use the gcloud CLI or REST API.

Console

  1. In the Google Cloud console, go to the Reservations page.

    Go to Reservations

  2. In the Name column, click the name of a reservation. A page that gives the details of the reservation appears.

  3. Click Run maintenance, and then select one of the following options:

    • To start maintenance on all blocks, select All capacity.

    • To start maintenance only on blocks with running VMs, select In-use capacity.

    • To start maintenance only on unused blocks and blocks with stopped or suspended VMs, select Unused capacity.

  4. To confirm, click Ok.

gcloud

Replace the following:

  • RESERVATION_NAME: the name of the reservation.

  • RESERVATION_MAINTENANCE_SCOPE: the maintenance scope for the reservation. Specify one of the following values:

    • To start maintenance on all blocks: all

    • To start maintenance only on blocks with running VMs: running

    • To start maintenance only on unused blocks and blocks with stopped or suspended VMs: unused

  • BLOCK_NAME: the name of a block within the reservation.

  • BLOCK_MAINTENANCE_SCOPE: the maintenance scope for the reservation block. Specify one of the following values:

    • To start maintenance on all hosts: all

    • To start maintenance only on hosts with running VMs: running

    • To start maintenance only on unused hosts and hosts with stopped or suspended VMs: unused

  • ZONE: the zone where the reservation exists.

REST

  • To start maintenance on a reservation, make a POST request to the beta reservations.performMaintenance method:

    POST https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/zones/ZONE/reservations/RESERVATION_NAME/performMaintenance
    
    {
      "maintenanceScope": "RESERVATION_MAINTENANCE_SCOPE"
    }
    
  • To start maintenance on a reservation block, make a POST request to the beta reservationBlocks.performMaintenance method:

    POST https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/zones/ZONE/reservations/RESERVATION_NAME/reservationBlocks/BLOCK_NAME/performMaintenance
    
    {
      "maintenanceScope": "BLOCK_MAINTENANCE_SCOPE"
    }
    

Replace the following:

  • PROJECT_ID: the ID of the project where Compute Engine automatically created the reservation.

  • ZONE: the zone where the reservation exists.

  • RESERVATION_NAME: the name of the reservation.

  • RESERVATION_MAINTENANCE_SCOPE: the maintenance scope for the reservation. Specify one of the following values:

    • To start maintenance on all blocks: ALL

    • To start maintenance only on blocks with running VMs: RUNNING

    • To start maintenance only on unused blocks and blocks with stopped or suspended VMs: UNUSED

  • BLOCK_NAME: the name of a block within the reservation.

  • BLOCK_MAINTENANCE_SCOPE: the maintenance scope for the reservation block. Specify one of the following values:

    • To start maintenance on all hosts: ALL

    • To start maintenance only on hosts with running VMs: RUNNING

    • To start maintenance only on unused hosts and hosts with stopped or suspended VMs: UNUSED

What's next