Work with cross-data-center replication

Kubernetes

1. Ensure external connectivity is enabled on your AlloyDB Omni primary database cluster. If external connectivity is not enabled, add the following to the spec section of the database cluster manifest:

     kind: DBCluster
     spec:
      allowExternalIncomingTraffic: true
   

2. To enable replication on your primary database cluster, apply a manifest similar to the following to your Kubernetes cluster on the primary data center:

    apiVersion: v1
    kind: Secret
    metadata:
      name: ha-rep-pw-DB_CLUSTER_NAME
    type: Opaque
    data:
      rep-user-pw: "ENCODED_PASSWORD"
    ---
    apiVersion: alloydbomni.dbadmin.goog/v1
    kind: Replication
    metadata:
      name: REPLICATION_NAME
    spec:
      dbcluster:
        name: DB_CLUSTER_NAME
      upstream:
        password:
          name: ha-rep-pw-DB_CLUSTER_NAME
   

   Replace the following:

   • DB_CLUSTER_NAME: The name of the database cluster—for example, dbc-1.
   • ENCODED_PASSWORD: The password for the database user to be used for replication from secondary databases, encoded as a base64 string—for example, Q2hhbmdlTWUxMjM= for ChangeMe123. The default value is alloydbreplica.
   • REPLICATION_NAME: the name of the replication—for example, replication-1.

   Wait for the replication status to be ready.

3. To get the upstream connection information that is used to configure replication on the secondary database cluster, run the following command:

     kubectl get replication REPLICATION_NAME
     kubectl get replication REPLICATION_NAME -o json | jq .status.upstream

   Sample output looks similar to the following:

     {
      "host": "35.230.32.36",
      "password": {
        "name": "ha-rep-pw-dbc-1"
      },
      "port": 5432,
      "replicationSlotName": "dbc_1_replication_1",
      "username": "alloydbreplica"
     }
   

4. Take a note of the output since you need it to enable replication on the secondary database cluster in the following step.

5. Create an AlloyDB Omni cluster on your Kubernetes cluster in the secondary data center with configuration identical to your primary database cluster.

6. Ensure external connectivity is enabled on your AlloyDB Omni secondary database cluster.

7. If external connectivity is not enabled, add the following to the spec section of its manifest:

    allowExternalIncomingTraffic: true
   

8. To enable replication on your secondary database cluster, apply a manifest similar to the following to your Kubernetes cluster on the secondary data center:

     apiVersion: v1
     kind: Secret
     metadata:
       name: ha-rep-pw-SECONDARY_DB_CLUSTER_NAME
     type: Opaque
     data:
       rep-user-pw: "ENCODED_PASSWORD"
     ---
     apiVersion: alloydbomni.dbadmin.goog/v1
     kind: Replication
     metadata:
       name: SECONDARY_REPLICATION_NAME
     spec:
       dbcluster:
         name: SECONDARY_DB_CLUSTER_NAME
       downstream:
         host: PRIMARY_HOST
         port: PRIMARY_PORT
         username: alloydbreplica
         password:
           name: ha-rep-pw-SECONDARY_DB_CLUSTER_NAME
         replicationSlotName: PRIMARY_REPLICATION_SLOT
         control: setup
   

   Replace the following:

   • SECONDARY_DB_CLUSTER_NAME: The name of the secondary database cluster—for example, dbc-2.
   • ENCODED_PASSWORD: The password for the database user to be used for replication the primary database cluster, encoded as a base64 string—for example, Q2hhbmdlTWUxMjM= for ChangeMe123. The default value is alloydbreplica.
   • SECONDARY_REPLICATION_NAME: the name of the replication—for example, replication-2.
   • PRIMARY_HOST: the connection endpoint of the primary database cluster from the output in step 3 that the secondary database can access for replication.
   • PRIMARY_PORT: the connection port of the primary database cluster from the output in step 3 that the secondary database can access for replication.
   • PRIMARY_REPLICATION_SLOT: the name of the replication slot on the primary database cluster from the output in step 3 that the secondary database can use for replication.

Kubernetes

kubectl get dbcluster SECONDARY_DB_CLUSTER_NAME
kubectl get replication SECONDARY_REPLICATION_NAME

Kubernetes

• Check the replication status of the secondary database cluster, to ensure that it's both ready and healthy.

  kubectl get replication SECONDARY_REPLICATION_NAME

• Stop all writes to the primary database cluster. Run the following query on your primary database cluster to check the replication lag of the secondary database. Confirm that the result shows a minimal lag.

  A lag value of 0 is ideal. If the lag is greater than 0, you can still promote the secondary database cluster, at the risk of losing some recent transactions already committed on the primary database cluster.

   psql -h PRIMARY_HOST -U postgres -d postgres -c 'SELECT application_name, pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag FROM pg_stat_replication;'

Kubernetes

    apiVersion: v1
    kind: Secret
    metadata:
      name: ha-rep-pw-SECONDARY_DB_CLUSTER_NAME
    type: Opaque
    data:
      rep-user-pw: "ENCODED_PASSWORD"
    ---
    apiVersion: alloydbomni.dbadmin.goog/v1
    kind: Replication
    metadata:
      name: SECONDARY_REPLICATION_NAME
    spec:
      dbcluster:
        name: SECONDARY_DB_CLUSTER_NAME
      downstream:
        host: PRIMARY_HOST
        port: PRIMARY_PORT
        username: alloydbreplica
        password:
          name: ha-rep-pw-SECONDARY_DB_CLUSTER_NAME
        replicationSlotName: PRIMARY_REPLICATION_SLOT
        control: promote

Kubernetes

• Check the replication status of the secondary database cluster, to ensure that it's both ready and healthy.

  kubectl get replication SECONDARY_REPLICATION_NAME

• Stop all writes to the primary database cluster. Run the following query on your primary database cluster to check the replication lag of the secondary database. Confirm that the result shows a lag value of 0.

   psql -h PRIMARY_HOST -U postgres -d postgres -c 'SELECT application_name, pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag FROM pg_stat_replication;'

Kubernetes

1. To convert your AlloyDB Omni secondary database cluster into a primary database cluster, update its replication manifest on your Kubernetes cluster in the secondary data center as follows:

      apiVersion: v1
      kind: Secret
      metadata:
       name: ha-rep-pw-SECONDARY_DB_CLUSTER_NAME
      type: Opaque
      data:
        rep-user-pw: "ENCODED_PASSWORD"
      ---
      apiVersion: alloydbomni.dbadmin.goog/v1
      kind: Replication
      metadata:
       name: SECONDARY_REPLICATION_NAME
      spec:
       dbcluster:
          name: SECONDARY_DB_CLUSTER_NAME
        upstream:
          password:
            name: ha-rep-pw-SECONDARY_DB_CLUSTER_NAME
   

   Wait for the replication status to be ready.

2. To get the upstream connection information for replication, run the following command:

   kubectl get replication SECONDARY_REPLICATION_NAME
   kubectl get replication SECONDARY_REPLICATION_NAME -o json | jq .status.upstream

   Sample output looks similar to the following:

    {
      "host": "34.23.207.137",
      "password": {
        "name": "ha-rep-pw-dbc-2"
      },
      "port": 5432,
      "replicationSlotName": "dbc_2_replication_2",
      "username": "alloydbreplica"
    }
   

3. To convert your AlloyDB Omni primary database cluster into a secondary database cluster, update its replication manifest on your Kubernetes cluster on the primary data center similar to the following:

       apiVersion: v1
       kind: Secret
       metadata:
        name: ha-rep-pw-DB_CLUSTER_NAME
       type: Opaque
       data:
         rep-user-pw: "ENCODED_PASSWORD"
       ---
       apiVersion: alloydbomni.dbadmin.goog/v1
       kind: Replication
       metadata:
         name: REPLICATION_NAME
       spec:
         dbcluster:
           name: DB_CLUSTER_NAME
         downstream:
           host: SECONDARY_HOST
           port: SECONDARY_PORT
           username: alloydbreplica
           password:
             name: ha-rep-pw-DB_CLUSTER_NAME
           replicationSlotName: SECONDARY_REPLICATION_SLOT
          control: rewind
   

   Wait for the replication status to become ready and healthy.

4. To verify the replication status use:

   kubectl get replication REPLICATION_NAME