Perform a minor database version upgrade for AlloyDB Omni on Kubernetes

This page describes how to perform a minor database version upgrade for AlloyDB Omni on Kubernetes.

To perform a minor database version upgrade, there are two options:

  • Low downtime upgrade: For high availability (HA) environments running AlloyDB Omni version 15.7.1 or later, AlloyDB Omni upgrades your standby instances first. Then, the AlloyDB Omni operator performs a switchover, promoting one of the upgraded standby instances to be your new primary instance. After the switchover succeeds, your old primary instance is updated.

    This process ensures minimal downtime during the upgrade.

  • Simultaneous upgrade: For all other circumstances, the AlloyDB Omni operator upgrades all instances simultaneously. This means you will experience downtime during the upgrade.

Limitations

For low downtime upgrades, one standby instance is unavailable at any given time. To ensure that your database cluster doesn't reach zero Recovery Point Objective (RPO) and doesn't risk losing data, your database cluster must have one primary instance and at least two standby instances.

Before your begin

  • If your cluster is HA and the AlloyDB Omni version is older than 15.7.1, follow the steps listed in Update the database clusters before following this minor version upgrade process.
  • Identify a low traffic period where you can perform the minor version upgrade.
  • To avoid any data loss, back up your data.

Enable the low downtime minor database version upgrade process

To enable the low downtime minor database version upgrade process, add the following annotation to your database cluster.

kubectl annotate dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME
dbcluster.dbadmin.goog/enableLDTM=true

Replace the following variable:

  • DB_CLUSTER_NAME: the name of your database cluster. It's the same database cluster name that you provided when you created it. For more information, see Install AlloyDB Omni on Kubernetes.

Upgrade your AlloyDB Omni version

To upgrade your 16.8.0 version, update the databaseVersion and the controlPlaneAgentsVersion fields in the cluster's manifest file, and then re-apply the file.

The following is the beginning of a manifest file that specifies version 16.8.0 for databaseVersion and version 1.5.0 for controlPlaneAgentsVersion:

apiVersion: alloydbomni.dbadmin.goog/v1
kind: DBCluster
metadata:
    name: DB_CLUSTER_NAME
spec:
    databaseVersion: "16.8.0"
    controlPlaneAgentsVersion: "1.5.0"
...

Replace the following variable:

  • DB_CLUSTER_NAME: the name of your database cluster. It's the same database cluster name that you provided when you created it. For more information, see Install AlloyDB Omni on Kubernetes.

Monitor the upgrade process

After you update your manifest file, the AlloyDB Omni operator starts the upgrade process. To monitor the upgrade process, check the DBCUpgradeInProgress condition.

kubectl get dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME -o yaml | yq '.status.conditions[] | select(.type == "DBCUpgradeInProgress")'

Replace the following variable:

  • DB_CLUSTER_NAME: the name of your database cluster. It's the same database cluster name that you provided when you created it. For more information, see Install AlloyDB Omni on Kubernetes.

While the process is in progress, the status is true. When the process is complete, the condition's status changes to false.

Troubleshooting

If you receive any failure messages during the upgrade process, see the following sections:

Pre-upgrade failures

If you receive a pre-upgrade failure on your database cluster, then check the message and address the problem accordingly.

If you would like to bypass the pre-upgrade failure message, then you can enable the force-upgrade annotation.

kubectl annotate dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME upgrade.alloydbomni.dbadmin.google/force-upgrade=true

Replace the following variable:

  • DB_CLUSTER_NAME: the name of your database cluster. It's the same database cluster name that you provided when you created it. For more information, see Install AlloyDB Omni on Kubernetes.

After the upgrade process completes, set the force-upgrade annotation to false.

Upgrade failures

During the automatic upgrade process, there are several points where it might fail in HA environments. For more information about each failure scenario and which subsequent actions the AlloyDB Omni operator takes, see the following table.

Failure message Description Required user actions
standby instance upgrade succeeded but the replication lag of the standby(s) is too high to be promoted to synchronous standby(s)

The upgrade process succeeded, but the standby instance didn't catch up to the primary instance to establish synchronous replication.

A high amount of traffic goes to the primary instance. As the traffic decreases, the standby instance catches up gradually. After this happens, the HAReady condition becomes true.

Choose an option in Fix primary and standby instances with different minor versions .

all standbys upgrade succeeded but the switchover instance failed to promote an upgraded standby.

Your standby instances upgraded successfully, but the switchover process failed.

  1. Check the status of the switchover custom resource to determine what caused the failure.
  2. Choose an option in Fix primary and standby instances with different minor versions .
standby instance upgrade failed but rollback succeeded.

Your standby instance didn't upgrade successfully, but the AlloyDB Omni operator restored it to its previous version successfully.

  1. Check the upgrade failure messages.
  2. Choose an option in Fix primary and standby instances with different minor versions .
standby instance upgrade failed and rollback failed.

Your standby instance didn't upgrade successfully and the AlloyDB Omni operator can't restore the instance to its previous version.

Contact Google support to troubleshoot.

Fix primary and standby instances with different minor versions

To resolve this problem, choose one of the following options:

  • If the issue that caused the upgrade failure is fixed, retry the upgrade.

    To retry the upgrade, remove the upgrade's start-time annotation from your instance. After you remove the annotation, the AlloyDB Omni operator generates a new start time and re-initiates the upgrade process.

    kubectl annotate dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME upgrade.alloydbomni.dbadmin.google/start-time-

    Replace the following variable:

    • DB_CLUSTER_NAME: the name of your database cluster. It's the same database cluster name that you provided when you created it. For more information, see Install AlloyDB Omni on Kubernetes.

  • If the issue that caused the upgrade failure persists, then downgrade your instance to the previous AlloyDB Omni operator version.

    To downgrade your instance, follow the upgrade process and change the databaseVersion and controlPlaneAgentsVersion fields in the manifest file to the version you used before.