Use maintenance mode on a database cluster in Kubernetes

If your database runs in a Kubernetes cluster, then you can put the database in maintenance mode by disabling liveness probes and startup probes, which run on your containerized application, to ensure that the database pod is in a healthy state. Maintenance mode helps you place a pod in a running state when you need to make a maintenance update or repair the pod.

Enable maintenance mode

1. Verify that probes exist on a database pod:

   1. Liveness probe:

      kubectl get po -n NAMESPACE DATABASE_POD_NAME  -o jsonpath='{.spec.containers[?(@.name=="database")].livenessProbe}'

      Replace the following:

      • NAMESPACE: the name of a namespace used to run your database—for example, db.
      • DATABASE_POD_NAME: the name of the database pod—for example, al-4017-dbcluster-sample-0.

      The following shows sample output with a liveness probe:

      {"failureThreshold":3,"httpGet":{"path":"/healthz?","port":8090,"scheme":"HTTP"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":3}
      

   2. Startup probe:

      kubectl get po -n NAMESPACE DATABASE_POD_NAME -o jsonpath='{.spec.containers[?(@.name=="database")].startupProbe}'

      The following shows sample output with a startup probe:

      {"exec":{"command":["/usr/lib/postgresql/15/bin/psql","-hlocalhost","-Ualloydbadmin","-c","SELECT 1"]},"failureThreshold":180,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1}
      

2. Set mode to maintenance in the spec section of your DBCluster manifest:

   apiVersion: alloydbomni.dbadmin.goog/v1
   kind: DBCluster
   metadata:
     name: DB_CLUSTER_NAME
     namespace: NAMESPACE
   spec:
     allowExternalIncomingTraffic: false
     isDeleted: false
     mode: maintenance
     primarySpec:
     ...
   

   Replace DB_CLUSTER_NAME with the name of this database cluster—for example, my-db-cluster.

3. Apply the manifest:

   kubectl apply -f DB_CLUSTER_YAML

   Replace DB_CLUSTER_YAML with the name of this database cluster manifest file—for example, alloydb-omni-db-cluster.yaml.

4. To verify that maintenance mode is enabled, run the following commands:

   • Liveness probe:

     kubectl get po -n NAMESPACE DATABASE_POD_NAME  -o jsonpath='{.spec.containers[?(@.name=="database")].livenessProbe}'

   • Startup probe:

     kubectl get po -n NAMESPACE DATABASE_POD_NAME  -o jsonpath='{.spec.containers[?(@.name=="database")].startupProbe}'

   The output doesn't contain the Liveness and Startup fields in the spec section of your POD_NAME pod.

Disable maintenance mode

To return to the standard database cluster behavior with running liveliness and startup probes, use the following command to remove maintenance from the spec section of your DBCluster manifest:

kubectl patch dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME -p '{"spec":{"mode":""}}' --type=merge -n DB_CLUSTER_NAMESPACE --type=merge

Sample outputs:

• With a liveness probe:

  {"failureThreshold":3,"httpGet":{"path":"/healthz?","port":8090,"scheme":"HTTP"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":3}
  

• With a startup probe:

  {"exec":{"command":["/usr/lib/postgresql/15/bin/psql","-hlocalhost","-Ualloydbadmin","-c","SELECT 1"]},"failureThreshold":180,  "periodSeconds":10,"successThreshold":1,"timeoutSeconds":1}