Change dual-region quorum

Monitoring REST API

Use the Monitoring projects.timeSeries.list API to query the Dual-region quorum health timeline to check if a region is healthy at a given time.

Before using any of the request data, make the following replacements:

• PROJECT_ID: your project ID.
• DUAL_REGION_LOCATION: the location of the region in the dual-region that you want to check.
• START_TIME: the start time of the query. We recommend using 5 minutes before the current time.
• END_TIME: the end time of the query. We recommend using the current time.

HTTP method and URL:

GET https://monitoring.googleapis.com/v3/projects/PROJECT_ID/timeSeries

Request JSON body:

{
  "name": "PROJECT_ID",
  "aggregation.alignmentPeriod": "60s",
  "aggregation.crossSeriesReducer": "REDUCE_FRACTION_TRUE",
  "aggregation.groupByFields": "resource.labels.location",
  "aggregation.perSeriesAligner": "ALIGN_NEXT_OLDER",
  "filter": "metric.labels.quorum_availability = "Healthy" AND metric.type = "spanner.googleapis.com/instance/dual_region_quorum_availability" AND resource.labels.location = "DUAL_REGION_LOCATION"",
  "interval.startTime": "START_TIME",
  "interval.endTime": "END_TIME"
}

To send your request, expand one of these options:

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://monitoring.googleapis.com/v3/projects/PROJECT_ID/timeSeries"

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://monitoring.googleapis.com/v3/projects/PROJECT_ID/timeSeries" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{
  "timeSeries": [
    {
      "metric": {
        "type": "spanner.googleapis.com/instance/dual_region_quorum_availability"
      },
      "resource": {
        "type": "spanner_instance",
        "labels": {
          "project_id": "spanner-project",
          "location": "australia-southeast1"
        }
      },
      "metricKind": "GAUGE",
      "valueType": "DOUBLE",
      "points": [
        {
          "interval": {
            "startTime": "2024-07-11T05:41:23Z",
            "endTime": "2024-07-11T05:41:23Z"
          },
          "value": {
            "doubleValue": 1
          }
        }
      ]
    }
  ],
  "unit": "10^2.%"
}

If you don't see a similar response, your region might not be healthy, and you might need to change the dual-region quorum from dual-region to single region.

gcloud CLI

1. Download the dual-region-quorum-health-check-script.sh file.

   This bash script checks the region health of your single region. If the region is healthy, then the script runs the gcloud spanner databases change-quorum command to manually failover the dual-region quorum from dual-region to single region.

2. Replace the following variables in the script:

   • PROJECT: your project ID.
   • INSTANCE: your instance ID.
   • DATABASE: your database ID.
   • SERVING_LOCATION: the location of the region in the dual-region that you want to check.

3. Run the script in the development environment of your choice. For more information, see Install the Google Cloud CLI and set up the Spanner API.

4. Manually failback if your region was unhealthy and there was a failover.

Google Cloud console

1. Go to the Spanner Instances page in the Google Cloud console.

   Go to the Instances page

2. Click the name of the instance that is in a dual-region configuration.

3. Click System insights in the navigation menu.

4. Find the Dual-region quorum health timeline metric.

5. If the Dual-region quorum health timeline shows there is disruption in a region, click Change region quorum.

   The Cloud Shell opens.

6. To change the dual-region quorum from dual-region to a single region, enter the following command found in the gcloud CLI tab: gcloud spanner databases change-quorum. For detailed instructions, refer to the gcloud tab.

gcloud CLI

Use the gcloud spanner databases change-quorum command to change the dual-region quorum from dual-region to a single region.

gcloud spanner databases change-quorum
    DATABASE_ID --instance=INSTANCE_ID
    --single-region --serving-location=SERVING_LOCATION
    [--etag=ETAG]

Replace the following:

• DATABASE_ID: The permanent identifier of the database.

• INSTANCE_ID: The permanent identifier for the instance.

• SERVING_LOCATION: The regional instance configuration to which you want to failover. For example, if asia-south1 (Mumbai) is unhealthy, and you want to failover to asia-south2 (Delhi), input asia-south2. Make sure the SERVING_LOCATION is the healthy region. Choosing a wrong region to failover results in database unavailability, which is unrecoverable before the region is back online.

Optional flags:

• --etag=ETAG: The ETAG argument can be used for replay protection.

To check the status of your quorum change operation, run the gcloud spanner databases describe command. The quorumInfo field provides information about the operation.

gcloud spanner databases describe DATABASE_ID
    --instance=INSTANCE_ID

Google Cloud console

1. Go to the Spanner Instances page in the Google Cloud console.

   Go to the Instances page

2. Click the name of the instance that is in a dual-region configuration.

3. Click System insights in the navigation menu.

4. Find the Dual-region quorum health timeline metric.

5. On the Dual-region quorum health timeline, click Change region quorum.

   The Cloud Shell opens.

6. To change the dual-region quorum from dual-region to a single region, enter the following command found in the gcloud CLI tab: gcloud spanner databases change-quorum. For detailed instructions, refer to the gcloud tab.

gcloud CLI

Use the gcloud spanner databases change-quorum command to change the dual-region quorum from single region to dual-region.

gcloud spanner databases change-quorum
    DATABASE_ID --instance=INSTANCE_ID
    --dual-region
    [--etag=ETAG]

Replace the following:

• DATABASE_ID: The permanent identifier of the database.

• INSTANCE_ID: The permanent identifier for the instance.

Optional flags:

• --etag=ETAG: The ETAG argument can be used for optimistic concurrency control.

To check the status of your quorum change operation, run the gcloud spanner databases describe command. The quorumInfo field provides information about the operation.

gcloud spanner databases describe DATABASE_ID
    --instance=INSTANCE_ID