Rotating Cassandra credentials in Kubernetes secrets

Rotating Cassandra secrets in Kubernetes secrets

This procedure describes rotating Cassandra credentials in Kubernetes secrets in your cluster. For rotating credentials within Hashicorp Vault, see Rotating Cassandra credentials in Hashicorp Vault.

This procedure will enable you to rotate the Cassandra credentials used for production Apigee Hybrid services without causing downtime by automating and triggering three tasks:

Rotation of user credentials
Rollback of the rotation
Cleanup of old and new credentials

Rotate Cassandra secrets in Kubernetes secrets

There are six username-password pairs of Cassandra credentials you can rotate:

default (password rotation only)
admin_user
ddl_user
dml_user
jmxuser
apigee

Note: You can only rotate the password for the default Cassandra user.You cannot change the username for the Default Cassandra user.

Prepare for credential rotation

To perform these steps, you must have permission to run kubectl apply -f within the cluster.

Backup the Cassandra database. This backup is to ensure recovery is possible to pre-rotated credentials. See Cassandra backup overview.
Make sure the cluster is in a healthy state, that all Apigee resources are running and no state changes are pending.

Single region credential rotation

Create a new Kubernetes secret in your Apigee namespace which holds the Cassandra user credentials you want to rotate, following Setting usernames and passwords in a Kubernetes secret.
Perform the rotation precheck.
1. Create a new SecretRotation custom resource using the following template:
```
# rotation-k8s-secrets.yaml

apiVersion: apigee.cloud.google.com/v1alpha1
kind: SecretRotation
metadata:
  name: ROTATION_NAME
  namespace: APIGEE_NAMESPACE
spec:
  organizationId: APIGEE_ORG
  rotationId: ROTATION_ID
  timeoutMinutes: TIMEOUT_MINUTES # optional. 480 recommended.
  precheck: true
  cassandra:
    oldSecretRef: OLD_SECRET_REF
    newSecretRef: NEW_SECRET_REF
    jobType: ROTATE
```
  Where:
  - ROTATION_NAME: A name for the rotation job, for example: sr-1-precheck.
  - APIGEE_NAMESPACE: your Apigee namespace.
  - APIGEE_ORG: Your Apigee organization ID.
  - ROTATION_ID: A custom identifier, for example: rot-1-precheck.
  - TIMEOUT_MINUTES: Optional. Overrides the default (480m == 8hr). <=0 means infinite timeout.
  - OLD_SECRET_REF: The secret name currently being used by the apigeedatastore. You can find this using the following command:
```
kubectl -n APIGEE_NAMESPACE get apigeedatastore default -o jsonpath='{.spec.credentialRef}'
```
  - NEW_SECRET_REF: The secret name created in the previous step.
2. Trigger the rotation precheck job by applying the rotation YAML file:
```
kubectl -n APIGEE_NAMESPACE apply -f ROTATION_YAML_FILE
```
3. Check the job status and wait for it to complete.
```
kubectl -n APIGEE_NAMESPACE describe sr ROTATION_NAME
```
  When the job is finished, the output will contain:
```
Status:
  Message:  rotation complete
  State:    complete
```
  Tip: You can also view the logs while the job is running with the following command:
```
kubectl -n APIGEE_NAMESPACE logs sr-ROTATION_ID-rotate-job-HASH
```
When the rotation precheck job completes, perform the rotation:
1. Make the following changes in the rotation YAML file:
  - Change the metadata.name to a name indicating this is not a precheck, for example: sr-1.
  - Set spec.precheck to false.
  - Change spec.rotationId to a new identifier, for example: rot-1.
2. Apply the file again to perform the rotation.
```
kubectl -n APIGEE_NAMESPACE apply -f ROTATION_YAML_FILE
```
Once the rotation job completes, validate that traffic is still flowing correctly.
After validating your traffic, perform the cleanup of the old secrets.
Caution: Once you trigger the cleanup job, you will not be able to perform a rollback to revert the rotation process and leave the cluster in its original state. See Rolling back a rotation.
1. Make the following changes in the rotation YAML file:
  - Change the metadata.name to a name indicating this is a cleanup job, for example: sr-1-cleanup.
  - Set spec.cassandra.jobType to CLEANUP.
2. Apply the file again to perform the cleanup.
```
kubectl -n APIGEE_NAMESPACE apply -f ROTATION_YAML_FILE
```
3. Check the job status and wait for it to complete.
```
kubectl -n APIGEE_NAMESPACE describe sr ROTATION_NAME
```
  When the cleanup job is completed, the rotation process is complete.
Take a backup of the Cassandra database. This backup is to ensure recovery is possible to post-rotated credentials.
Delete the old Cassandra credential secret.

Multi-region credential rotation

in each region, create a new Kubernetes secret in the your Apigee namespace which holds the Cassandra user credentials you want to rotate, following Setting usernames and passwords in a Kubernetes secret.
The credentials must be the same in each region.
Perform the rotation precheck.
1. Create a new SecretRotation custom resource using the following template:
```
# rotation-k8s-secrets.yaml

apiVersion: apigee.cloud.google.com/v1alpha1
kind: SecretRotation
metadata:
  name: ROTATION_NAME
  namespace: APIGEE_NAMESPACE
spec:
  organizationId: APIGEE_ORG
  rotationId: ROTATION_ID
  timeoutMinutes: TIMEOUT_MINUTES # optional. 480 recommended.
  precheck: true
  cassandra:
    oldSecretRef: OLD_SECRET_REF
    newSecretRef: NEW_SECRET_REF
    jobType: ROTATE
```
  Where:
  - ROTATION_NAME: A name for the rotation job, for example: sr-1-precheck.
  - APIGEE_NAMESPACE: your Apigee namespace.
  - APIGEE_ORG: Your Apigee organization ID.
  - ROTATION_ID: A custom identifier, for example: rot-1-precheck.
  - TIMEOUT_MINUTES: Optional. Overrides the default (480m == 8hr). <=0 means infinite timeout.
  - OLD_SECRET_REF: The secret name currently being used by the apigeedatastore. You can find this using the following command:
```
kubectl -n APIGEE_NAMESPACE get apigeedatastore default -o jsonpath='{.spec.credentialRef}'
```
  - NEW_SECRET_REF: The secret name created in the previous step.
2. In the first region, trigger the rotation precheck job by applying the rotation YAML file:
```
kubectl -n APIGEE_NAMESPACE apply -f ROTATION_YAML_FILE
```
3. Check the job status and wait for it to complete.
```
kubectl -n APIGEE_NAMESPACE describe sr ROTATION_NAME
```
  When the job is finished, the output will contain:
```
Status:
  Message:  rotation complete
  State:    complete
```
  Tip: You can also view the logs while the job is running with the following command:
```
kubectl -n APIGEE_NAMESPACE logs sr-ROTATION_ID-rotate-job-HASH
```
When the rotation precheck job completes, perform the rotation in the same region:
1. Make the following changes in the rotation YAML file:
  - Change the metadata.name to a name indicating this is not a precheck, for example: sr-1.
  - Set spec.precheck to false.
  - Change spec.rotationId to a new identifier, for example: rot-1.
2. Apply the file again to perform the rotation.
```
kubectl -n APIGEE_NAMESPACE apply -f ROTATION_YAML_FILE
```
Once the rotation job completes, validate that traffic is still flowing correctly.
Repeat the rotation steps for the remaining regions.
After validating your traffic, perform the cleanup of the old secrets. You only need to perform the cleanup in a single region. It does not matter which region you choose.
Caution: Once you trigger the cleanup job, you will not be able to perform a rollback to revert the rotation process and leave the cluster in its original state. See Rolling back a rotation.
1. Make the following changes in the rotation YAML file:
  - Change the metadata.name to a name indicating this is a cleanup job, for example: sr-1-cleanup.
  - Set spec.cassandra.jobType to CLEANUP.
2. Apply the file again to perform the cleanup.
```
kubectl -n APIGEE_NAMESPACE apply -f ROTATION_YAML_FILE
```
3. Check the job status and wait for it to complete.
```
kubectl -n APIGEE_NAMESPACE describe sr ROTATION_NAME
```
  When the cleanup job is completed, the rotation process is complete.
Take a backup of the Cassandra database. This backup is to ensure recovery is possible to post-rotated credentials.
Delete the old Cassandra credential secret.

Rolling back a rotation

Perform the following steps in each region.

Create a new SecretRotation custom resource using the following template:
```
# rollback-k8s-secrets.yaml

apiVersion: apigee.cloud.google.com/v1alpha1
kind: SecretRotation
metadata:
  name: ROLLBACK_NAME
  namespace: APIGEE_NAMESPACE
spec:
  organizationId: APIGEE_ORG
  rotationId: ROTATION_ID # match the current rotation.
  timeoutMinutes: TIMEOUT_MINUTES # optional.
  precheck: false
  cassandra:
    oldSecretRef: OLD_SECRET_REF # Must match the previous oldSecretRef.
    newSecretRef: NEW_SECRET_REF # Must match the previous newSecretRef.
    jobType: ROLLBACK
```
Where:
- ROLLBACK_NAME: A name for the rollback job, for example: sr-1-rollback.
- APIGEE_NAMESPACE: your Apigee namespace.
- APIGEE_ORG: Your Apigee organization ID.
- ROTATION_ID: The id of the current rotation that you are rolling back, for example: rot-1.
- TIMEOUT_MINUTES: Optional. Overrides the default (480m == 8hr). <=0 means infinite timeout.
- OLD_SECRET_REF: This must match the secret name for oldSecretRef: in the rotation YAML file you used in Single region rotation or Multi-region rotation procedure.
- NEW_SECRET_REF: this must match the secret name for newSecretRef: in the rotation YAML file you used in Single region rotation or Multi-region rotation procedure.

Apply the rollback:

kubectl -n APIGEE_NAMESPACE apply -f ROLLBACK_YAML_FILE

Check the job status and wait for it to complete.

kubectl -n APIGEE_NAMESPACE describe sr ROTATION_NAME

When the rollback(s) complete, verify that traffic is still flowing correctly.
When the traffic is flowing correctly, repeat the rollback process in each remaining region.
Once you have completed the rollback and verified that traffic is still flowing correctly, start the cleanup process. You only need to perform the cleanup in a single region. It does not matter which region you choose.
Make the following changes in the rotation YAML file:
- Change metadata.name to a name indicating this is a cleanup job, for example: sr-1-cleanup-rollback.
- Change spec.cassandra.jobType to CLEANUP_ROLLBACK.

Apply the file to trigger the cleanup job:

kubectl -n APIGEE_NAMESPACE apply -f ROTATION_YAML_FILE