Manage and monitor AlloyDB Omni

Kubernetes

Find your database cluster log files

postgresql.audit and postgresql.log files are located on the file system of the database pod. postgresql.audit is only present if you enabled pgaudit.

To access these files, follow these steps:

1. Define an environment variable containing the name of the database pod.

   export DB_POD=`kubectl get pod -l alloydbomni.internal.dbadmin.goog/dbcluster=DB_CLUSTER_NAME,alloydbomni.internal.dbadmin.goog/task-type=database -o jsonpath='{.items[0].metadata.name}'`

   Replace DB_CLUSTER_NAME with the name of your database cluster. It's the same database cluster name you declared when you created it.

2. Run a shell on the database pod as root.

   kubectl exec ${DB_POD} -it -- /bin/bash

3. Find the log files in the /obs/diagnostic/ directory:

   • /obs/diagnostic/postgresql.audit
   • /obs/diagnostic/postgresql.log

List monitoring services

kubectl get svc -n NAMESPACE | grep monitoring

al-1060-dbc-monitoring-system   ClusterIP   10.0.15.227   <none>        9187/TCP   7d20h
al-3de6-dbc-monitoring-system   ClusterIP   10.0.5.205    <none>        9187/TCP   7d19h
al-4bc0-dbc-monitoring-system   ClusterIP   10.0.15.92    <none>        9187/TCP   7d19h

kubectl get svc -n NAMESPACE | grep monitoring

al-2953-dbcluster-foo7-monitoring-db           ClusterIP   10.36.3.243    <none>        9187/TCP   44m
al-2953-dbcluster-foo7-monitoring-system       ClusterIP   10.36.7.72     <none>        9187/TCP   44m

View Prometheus metrics from the command line

The port 9187 is named as metricsalloydbomni for all monitoring services.

1. Set up port forwarding from your local environment to the monitoring service.

   kubectl port-forward service/MONITORING_SERVICE -n NAMESPACE MONITORING_METRICS_PORT:metricsalloydbomni
   

   Replace the following:

   • MONITORING_SERVICE: The name of the monitoring service that you want to forward—for example, al-1060-dbc-monitoring-system.

   • NAMESPACE: The namespace where your cluster belongs.

   • MONITORING_METRICS_PORT: A local available TCP port.

   The following response shows that the services are being forwarded.

   Forwarding from 127.0.0.1:9187 -> 9187
   Forwarding from [::1]:9187 -> 9187
   

2. While the previous command runs, you can access monitoring metrics through HTTP on the port that you specified. For example, you can use curl to see all of the metrics as plain text:

   curl http://localhost:MONITORING_METRICS_PORT/metrics
   

View metrics using the Prometheus API

The alloydbomni.internal.dbadmin.goog/task-type label key and the metricsalloydbomni port is available as a default for all monitoring services in AlloyDB Omni. You can use them together with a single serviceMonitor custom resource to select all the services for all namespaces in your database cluster.

For more information about using the Prometheus API, see the Prometheus Operator documentation.

The following is an example spec field of the serviceMonitor custom resource that includes the alloydbomni.internal.dbadmin.gdc.goog/task-type label key and metricsalloydbomni port. The serviceMonitor custom resource monitors and collects all the kubernetes services in all namespaces

For more information about the complete ServiceMonitor definition, see the ServiceMonitor custom resource definition .

    spec:
      selector:
        matchLabels:
          alloydbomni.internal.dbadmin.goog/task-type: monitoring
      namespaceSelector:
        any: true
      endpoints:
        - port: metricsalloydbomni

    spec:
      selector:
        matchExpressions:
        - key: alloydbomni.internal.dbadmin.gdc.goog/task-type
          operator: Exists
          values: []
      namespaceSelector:
        any: true
      endpoints:
      - port: metricsalloydbomni

View metrics using Grafana

For a visual representation of metrics in AlloyDB Omni on Kubernetes, use the monitoring dashboard. The monitoring dashboard relies on a basic observability stack that consists of Prometheus and Grafana. To configure the monitoring dashboard to collect metrics from AlloyDB Omni, follow these steps:

1. To download the Grafana dashboard, use the wget command:

   wget https://raw.githubusercontent.com/GoogleCloudPlatform/alloydb-omni-samples/refs/heads/main/monitoring-dashboards/grafana/alloydbomni_dashboard.yaml
   

2. You must download and install the grafana-operator before deploying Grafana in Kubernetes. For detailed instructions, see Installation.

3. Add the monitoring.dashboard/product=alloydb-omni label to the Grafana instance where you install the dashboard:

   kubectl label grafana/GRAFANA_INSTANCE_NAME monitoring.dashboard/product=alloydb-omni -n NAMESPACE
   

   Replace the following:

   • GRAFANA_INSTANCE_NAME: the name of the Grafana instance where you put the dashboard.
   • NAMESPACE: the namespace where you deployed the Grafana operator.

4. To apply the Grafana dashboard configuration to your AlloyDB Omni cluster on Kubernetes, use the following command:

   kubectl apply -f alloydbomni_dashboard.yaml -n NAMESPACE
   

   See Grafana operator documentation for information about using the Grafana operator.

5. To configure Grafana to use Prometheus as a data source, see Datasources.

6. To verify Grafana is configured correctly, do one of the following:

   • View the collection of Grafana panels on the AlloyDB Omni dashboard.

   • Retrieve information about the Grafana dashboard in a Kubernetes cluster:

     kubectl get grafanadashboard alloydb-omni-dashboard -n NAMESPACE -o jsonpath='{.status.conditions[?(@.type=="DashboardSynchronized")].status}'
     

     If the command returns True, the alloydb-omni-dashboard is successfully deployed to the Grafana instance.

Kubernetes

Delete your database cluster

To delete your database cluster, set isDeleted to true in its manifest. You can accomplish this with the following command.

kubectl patch dbclusters.alloydbomni.dbadmin.goog DB_CLUSTER_NAME -p '{"spec":{"isDeleted":true}}' --type=merge

Replace DB_CLUSTER_NAME with the name of your database cluster. It's the same database cluster name you declared when you created it.

Uninstall the AlloyDB Omni operator

To uninstall the AlloyDB Omni Kubernetes operator from your Kubernetes cluster, take the following steps:

1. Delete all of your database clusters:

   for ns in $(kubectl get dbclusters.alloydbomni.dbadmin.goog --all-namespaces -o=jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}'); do
   for cr in $(kubectl get dbclusters.alloydbomni.dbadmin.goog -n $ns -o=jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
   kubectl patch dbclusters.alloydbomni.dbadmin.goog $cr -n $ns --type=merge -p '{"spec":{"isDeleted":true}}'
   done
   done

2. Wait for the AlloyDB Omni Kubernetes operator to delete all of your database clusters. Use the following command to check whether any database resources remain:

   kubectl get dbclusters.alloydbomni.dbadmin.goog --all-namespaces

3. Delete other resources that the AlloyDB Omni Kubernetes operator created:

   kubectl delete failovers.alloydbomni.dbadmin.goog --all --all-namespaces
   kubectl delete restores.alloydbomni.dbadmin.goog --all --all-namespaces
   kubectl delete switchovers.alloydbomni.dbadmin.goog --all --all-namespaces

4. Uninstall the AlloyDB Omni Kubernetes operator:

   helm uninstall alloydbomni-operator --namespace alloydb-omni-system

5. Clean up secrets, custom resource descriptions, and namespaces related to the AlloyDB Omni Kubernetes operator:

   kubectl delete certificate -n alloydb-omni-system --all
   kubectl get secrets --all-namespaces -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,ANNOTATION:.metadata.annotations.cert-manager\.io/issuer-name | grep -E 'alloydbomni|dbs-al' | awk '{print $1 " " $2}' | xargs -n 2 kubectl delete secret -n
   kubectl delete crd -l alloydb-omni=true
   kubectl delete ns alloydb-omni-system