This document explains how to monitor, plan for, and perform pending maintenance on your reserved blocks of capacity in Hypercompute Cluster. To manage maintenance on your virtual machine (VM) instances, see instead Manage host events across VMs.
You can proactively manage upcoming maintenance host events on your reserved blocks of capacity, whether VMs are running on them or not. This approach helps you minimize disruptions and maintain optimal performance.
Before you begin
Select the tab for how you plan to use the samples on this page:
Console
When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.
gcloud
In the Google Cloud console, activate Cloud Shell.
At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
REST
To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.
After installing the Google Cloud CLI, initialize it by running the following command:
gcloud init
If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.
For more information, see Authenticate for using REST in the Google Cloud authentication documentation.
Required roles
To get the permissions that you need to manage host maintenance events across reservations, ask your administrator to grant you the following IAM roles:
-
Compute Admin (
roles/compute.admin
) on the project -
For read-only access to System Event audit logs:
Logs Viewer (
roles/logging.viewer
) on the project
For more information about granting roles, see Manage access to projects, folders, and organizations.
These predefined roles contain the permissions required to manage host maintenance events across reservations. To see the exact permissions that are required, expand the Required permissions section:
Required permissions
The following permissions are required to manage host maintenance events across reservations:
-
To start host maintenance for a reservation:
compute.reservations.performMaintenance
on the project -
To start host maintenance for a block within a reservation:
compute.reservationBlocks.performMaintenance
on the project -
To view the details of a reservation:
compute.reservations.get
on the project
You might also be able to get these permissions with custom roles or other predefined roles.
Overview
To optimize the maintenance of your reserved blocks of capacity, complete the following steps:
Set up notification alerts. Create log-based alerts to get notified about scheduled, started, or completed maintenance events for a reservation or a reservation block. This approach helps you proactively plan your activities and avoid unexpected downtime.
Manage maintenance across blocks of capacity. View and, if needed, manually start maintenance across your reservations or reservation blocks. This process helps you increase the resilience of your workloads to host errors, prevent downtime, and ensure that your applications remain available.
For more information about the frequency and maintenance behavior of your reserved machine types, see Understand host maintenance.
Set up notification alerts for reservations
You can get notified about scheduled, started, or completed maintenance events for a reservation or reservation block by creating log-based alerting policies.
To create an alert for the maintenance events of a reservation or a reservation block, complete the following procedure. Repeat this procedure for each alert that you want to create.
-
In the Google Cloud console, go to the Logs Explorer page:
If you use the search bar to find this page, then select the result whose subheading is Logging.
Click the Show query toggle to the on position.
In the Query pane, build one of the following queries. These queries filter log entries to identify specific maintenance events. Repeat this procedure for each query you want to create.
Receive alerts for maintenance of a reservation:
To receive alerts when maintenance is scheduled:
protoPayload.methodName="compute.reservations.upcomingGroupMaintenance" severity>=DEFAULT protoPayload.status.message =~ "scheduled"
To receive alerts when maintenance has completed:
protoPayload.methodName="compute.reservations.completedGroupMaintenance" severity>=DEFAULT protoPayload.status.message =~ "completed"
Receive alerts for maintenance of a block within a reservation:
To receive alerts when maintenance is scheduled:
protoPayload.methodName="compute.reservations.block.upcomingGroupMaintenance" severity>=DEFAULT protoPayload.status.message =~ "scheduled"
To receive alerts when maintenance has started:
protoPayload.methodName="compute.reservations.block.startGroupMaintenance" severity>=DEFAULT protoPayload.status.message =~ "started"
To receive alerts when maintenance has completed:
protoPayload.methodName="compute.reservations.block.completedGroupMaintenance" severity>=DEFAULT protoPayload.status.message =~ "completed"
To validate the query, click Run query. If the query is valid, then the Query results pane displays log entries that match the query.
In the Query results toolbar, click the Actions list, and then select
Create log alert. The Create logs-based alert policy pane appears.In the Alert details section, do the following:
In the Alert Policy Name field, enter a name for the policy.
In the Policy severity level list, select Warning (or a higher severity).
Click Next.
In the Choose logs to include in the alert section, click Next.
In the Set notification frequency and autoclose duration section, specify the following:
In the Time between notifications list, select how often you want to be notified.
In the Incident autoclose duration list, select after how long Cloud Logging stops sending notifications and automatically closes the incident.
Click Next.
In the Who should be notified? section, specify a notification channel for Logging to send notifications to.
Click Save.
Manage maintenance across reservations
You can view and control maintenance for your reservations or reservation blocks by doing one or more of the following:
To check the state and scheduled time of upcoming maintenance for your reservations or reservation blocks, view maintenance state.
To immediately start maintenance on a reservation or reservation block, rather than waiting for their scheduled maintenance time, manually start maintenance.
View maintenance state
You can view the upcoming maintenance state for a reservation and a reservation
block by checking the value of the upcomingGroupMaintenance
field in their
metadata. If a reservation lacks the upcomingGroupMaintenance
field, then no
host events are scheduled for the reservation or its blocks. For more
information about the fields in upcomingGroupMaintenance
, see
Maintenance status definitions
in the Compute Engine documentation.
You can view the maintenance state for multiple reservations simultaneously or single reservations. For multiple reservations, use the Google Cloud console or REST API. For single reservations, use any of the following options:
Console
In the Google Cloud console, go to the Reservations page.
In the Maintenance status column, Compute Engine displays the maintenance state of your reservations. If you don't see this column in the On-demand reservations table, then click
Column display options, select the Maintenance status checkbox, and then click OK.To view the maintenance state of a reservation block, complete the following steps:
In the Name column, click the name of the reservation. A page that gives the details of the reservation appears.
In the Blocks table, in the Maintenance column, Compute Engine displays the maintenance state of a reservation blocks.
gcloud
To view the maintenance state of a reservation, use the
gcloud beta compute reservations describe
command with the--flatten
flag set toreservationMaintenance
:gcloud beta compute reservations describe RESERVATION_NAME \ --flatten=reservationMaintenance \ --zone=ZONE
Replace the following:
RESERVATION_NAME
: the name of the reservation.ZONE
: the zone where the reservation exists.
If maintenance is scheduled for your reservation, then the output is similar to the following:
--- maintenanceOngoingCount: 0 maintenancePendingCount: 4 schedulingType: GROUPED upcomingGroupMaintenance: canReschedule: true maintenanceStatus: PENDING type: SCHEDULED
If maintenance isn't scheduled, then the output is similar to the following:
--- schedulingType: GROUPED
If maintenance is scheduled for your reservation, then you can view the maintenance state of the reservation blocks by using the
gcloud beta compute reservations blocks list
command:gcloud beta compute reservations blocks list RESERVATION_NAME \ --zone=ZONE
If maintenance is scheduled or ongoing for a block, then the block contains the
upcomingGroupMaintenance
field:--- name: example-fr-a3u-dense-1-block-0001 ... reservationMaintenance: maintenanceOngoingCount: 0 maintenancePendingCount: 2 schedulingType: GROUPED upcomingGroupMaintenance: canReschedule: true maintenanceStatus: PENDING type: SCHEDULED windowEndTime: '2025-05-13T14:00:00.000-08:00' windowStartTime: '2025-05-13T12:00:00.000-08:00' ... --- name: example-fr-a3u-dense-1-block-0002 ... reservationMaintenance: schedulingType: GROUPED ...
If the
schedulingType
field is set toINDEPENDENT
for a block, then theupcomingGroupMaintenance
field doesn't contain thewindowStartTime
andwindowEndTime
fields. To see when maintenance is scheduled for a VM running on a block, view the maintenance state for the VM.
REST
To view the maintenance state of your reservations, make a
GET
request to the betareservations.list
method. In the request URL, include thefields
andfilter
query parameters:GET https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/zones/ZONE/reservations?fields=items.name,items.machineType,items.reservationMaintenance:&filter=machineType%20eq%20%2E%2AMACHINE_TYPE
Replace the following:
PROJECT_ID
: the ID of the project where the reservations exist.ZONE
: the zone where the reservations exist.MACHINE_TYPE
: the reserved machine type in your reservations.
If maintenance is scheduled for a reservation, then the reservation contains the
upcomingGroupMaintenance
field, such as in the following output:{ "items": [ { "name": "reservation-01", "machineType": "https://www.googleapis.com/compute/beta/projects/example-project/zones/europe-west1-b/machineTypes/MACHINE_TYPE", "reservationMaintenance": { "maintenanceOngoingCount": 0, "maintenancePendingCount": 4, "schedulingType": "GROUPED", "upcomingGroupMaintenance": { "canReschedule": true, "maintenanceStatus": "PENDING" "type": "SCHEDULED" } } }, { "name": "reservation-02", "machineType": "https://www.googleapis.com/compute/beta/projects/example-project/zones/europe-west1-b/machineTypes/MACHINE_TYPE", "reservationMaintenance": { "schedulingType": "GROUPED" } } ] }
Optionally, to further narrow down a list of VMs, set the
filter
query parameter to a different filter expression.If maintenance is scheduled for your reservation, then you can view the maintenance state of the reservation blocks by making a
GET
request to the betareservationBlocks.get
method. In the request URL, include thefields
query parameter:GET https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/zones/ZONE/reservations/RESERVATION_NAME/reservationBlocks?fields=items.name,items.reservationMaintenance
Replace
RESERVATION_NAME
with the name of the reservation.If maintenance is scheduled for a block, then the block contains the
upcomingGroupMaintenance
field, such as in the following output:{ "items": [ { "name": "example-fr-a3u-dense-1-block-0001", "reservationMaintenance": { "maintenanceOngoingCount": 0, "maintenancePendingCount": 2, "schedulingType": "GROUPED", "upcomingGroupMaintenance": { "canReschedule": true, "maintenanceStatus": "PENDING", "type": "SCHEDULED", "windowStartTime": "2025-05-13T14:00:00.000-08:00", windowEndTime: "2025-05-13T12:00:00.000-08:00" } } }, { "name": "example-fr-a3u-dense-1-block-0002", "reservationMaintenance": { "schedulingType": "GROUPED" }, } ] }
If the
schedulingType
field is set toINDEPENDENT
for a block, then theupcomingGroupMaintenance
field doesn't contain thewindowStartTime
andwindowEndTime
fields. To see when maintenance is scheduled for a VM running on a block, view the maintenance state for the VM.
Manually start maintenance
You can manually start maintenance for your reservations or reservation blocks instead of waiting for the scheduled time.
Depending on the maintenance state of a reservation or reservation block, the following occurs:
Maintenance state | Description | What you see |
---|---|---|
Scheduled | Compute Engine has scheduled maintenance for the reservation. You can manually start maintenance before the scheduled time. |
|
In progress | Maintenance is underway. You can't reschedule it. |
|
Complete | Maintenance is finished. Compute Engine has removed all maintenance notifications from the VM. |
|
To manually start maintenance for a reservation or a reservation block, select one of the following options. To start maintenance on specific hosts within a reservation block, use the gcloud CLI or REST API.
Console
In the Google Cloud console, go to the Reservations page.
In the Name column, click the name of a reservation. A page that gives the details of the reservation appears.
Click Run maintenance, and then select one of the following options:
To start maintenance on all blocks, select All capacity.
To start maintenance only on blocks with running VMs, select In-use capacity.
To start maintenance only on unused blocks and blocks with stopped or suspended VMs, select Unused capacity.
To confirm, click Ok.
gcloud
To start maintenance on a reservation, use the
gcloud beta compute reservations perform-maintenance
command:gcloud beta compute reservations perform-maintenance RESERVATION_NAME \ --scope=RESERVATION_MAINTENANCE_SCOPE \ --zone=ZONE
To start maintenance on a reservation block, use the
gcloud beta compute reservations blocks perform-maintenance
command:gcloud beta compute reservations blocks perform-maintenance RESERVATION_NAME \ --block-name=BLOCK_NAME \ --scope=BLOCK_MAINTENANCE_SCOPE \ --zone=ZONE
Replace the following:
RESERVATION_NAME
: the name of the reservation.RESERVATION_MAINTENANCE_SCOPE
: the maintenance scope for the reservation. Specify one of the following values:To start maintenance on all blocks:
all
To start maintenance only on blocks with running VMs:
running
To start maintenance only on unused blocks and blocks with stopped or suspended VMs:
unused
BLOCK_NAME
: the name of a block within the reservation.BLOCK_MAINTENANCE_SCOPE
: the maintenance scope for the reservation block. Specify one of the following values:To start maintenance on all hosts:
all
To start maintenance only on hosts with running VMs:
running
To start maintenance only on unused hosts and hosts with stopped or suspended VMs:
unused
ZONE
: the zone where the reservation exists.
REST
To start maintenance on a reservation, make a
POST
request to the betareservations.performMaintenance
method:POST https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/zones/ZONE/reservations/RESERVATION_NAME/performMaintenance { "maintenanceScope": "RESERVATION_MAINTENANCE_SCOPE" }
To start maintenance on a reservation block, make a
POST
request to the betareservationBlocks.performMaintenance
method:POST https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/zones/ZONE/reservations/RESERVATION_NAME/reservationBlocks/BLOCK_NAME/performMaintenance { "maintenanceScope": "BLOCK_MAINTENANCE_SCOPE" }
Replace the following:
PROJECT_ID
: the ID of the project where Compute Engine automatically created the reservation.ZONE
: the zone where the reservation exists.RESERVATION_NAME
: the name of the reservation.RESERVATION_MAINTENANCE_SCOPE
: the maintenance scope for the reservation. Specify one of the following values:To start maintenance on all blocks:
ALL
To start maintenance only on blocks with running VMs:
RUNNING
To start maintenance only on unused blocks and blocks with stopped or suspended VMs:
UNUSED
BLOCK_NAME
: the name of a block within the reservation.BLOCK_MAINTENANCE_SCOPE
: the maintenance scope for the reservation block. Specify one of the following values:To start maintenance on all hosts:
ALL
To start maintenance only on hosts with running VMs:
RUNNING
To start maintenance only on unused hosts and hosts with stopped or suspended VMs:
UNUSED