Analyze AlloyDB Omni Kubernetes operator memory heap usage

This document describes how you can get a snapshot of the memory heap of AlloyDB Omni operator to help diagnose and debug potential memory issues. Use the following steps to get a memory heap for analysis:

  1. Determine which AlloyDB Omni operator deployment to analyze. The AlloyDB Omni operator includes two deployments, and you can analyze the memory heap for either one. To identify the two deployments, run the following command:

    kubectl get deployment -n alloydb-omni-system

    The output shows you the two deployments in the alloydb-omni-system namespace:

    • fleet-controller-manager
    • local-controller-manager

    You can get the memory heap of either deployment. For demonstration purposes, these steps show how to get the memory heap for the local-controller-manager deployment.

  2. Turn on memory analysis by specifying an available port to use. After the port is specified, the pod restarts. To specify an available port, use the pprof-address argument in the deployment:

    1. Open the deployment in a text editor by running the following command:

      kubectl edit -n alloydb-omni-system deploy local-controller-manager

    2. Specify the port in the args of the container section in the template spec:

      apiVersion: apps/v1
kind: Deployment
spec:
  ...
  template:
  ...
    spec:
      containers:
      - args
        - --pprof-address=:PORT

    3. Save the deployment file. After you save the deployment file, the pod restarts.

  3. Wait for the pod to restart before continuing to the next step.

    To make sure the pod restarted, run the following command:

    kubectl get pod -n alloydb-omni-system

    Verify that the output value in the pod's STATUS column is Running and the output value in its AGE column is a short duration. For example, if the STATUS column is Running and the value in the AGE column is 50s, then the pod has been running for 50 seconds after the restart.

  4. Turn on port forwarding using the following command:

    kubectl port-forward -n alloydb-omni-system DEPLOYMENT_POD_NAME PORT:PORT

    Replace DEPLOYMENT_POD_NAME with the name of your deployment as it appears in the NAME column of the output of the following command:

    kubectl get pod -n alloydb-omni-system

  5. In a different terminal, run the following command to create a file with a snapshot of the memory heap of the deployment:

    curl http://localhost:PORT/debug/pprof/heap > heap.out

  6. Save the heap.out file and use it to view the memory heap of the AlloyDB Omni operator deployment you chose to analyze.

  7. Turn off memory analysis by removing the pprof-address argument with the port you used from the deployment:

    1. Open the deployment in a text editor:

      kubectl edit -n alloydb-omni-system deploy local-controller-manager

    2. Remove the port --pprof-address=:PORT line from the args of the container section in template spec that you added earlier.

What's next