Storing data in an external secret manager

This guide tells you how to store and manage the following types of information in the external secret storage service, Hashicorp Vault, instead of in your overrides.yaml file.

AX Hash Salt
Redis password
Encryption keys

To store other kinds of information in Vault, see:

Prerequisites

A Kubernetes secrets store CSI driver. You can install it by following instructions in Secrets Store CSI Driver: Installation. For supported versions, see Apigee hybrid supported platforms and versions: Secret Store CSI driver.
The Vault CSI provider. You can install it by following instructions in Installing the Vault CSI provider. For supported versions, see Apigee hybrid supported platforms and versions: Vault.

Procedure

Create the Vault secrets, policies, and roles.

Use the Vault UI or APIs to create secrets and grant permissions for the Apigee Kubernetes service accounts to read those secrets as described here. The secrets need to consist of a key and one or more values as shown in the following table:

Secret Key	Secret Data
`secret/data/apigee/axhashsalt`	{ "ax-hash-salt": "`AX_HASH_SALT_VALUE`" }
`secret/data/apigee/redis`	{ "redis-password": "`REDIS_PASSWORD_VALUE`" }
`secret/data/apigee/orgencryptionkeys`	{ "kmsEncryptionKey": "`KMS_ENCRYPTION_KEY_VALUE`" "kvmEncryptionKey": "`KVM_ENCRYPTION_KEY_VALUE`" "contractEncryptionKey": "`CONTRACT_ENCRYPTION_KEY_VALUE`" } Important: The `contractEncryptionKey` values must be plaintext when created here. Do not base64 encode the value.
`secret/data/apigee/envencryptionkeys`	{ "cacheEncryptionKey": "`CACHE_ENCRYPTION_KEY_VALUE`" "kvmEncryptionKey": "`KVM_ENCRYPTION_KEY_VALUE`" "envKvmEncryptionKey": "`ENV_KVM_ENCRYPTION_KEY_VALUE`" "kmsEncryptionKey": "`KMS_ENCRYPTION_KEY_VALUE`" }

Within Vault, create policies which grant access to the secrets:

cat axhashsalt-auth-policy.txt
path "secret/data/apigee/axhashsalt" {
  capabilities = ["read"]
}
cat redis-auth-policy.txt
path "secret/data/apigee/redis" {
  capabilities = ["read"]
}
cat orgencryptionkeys-auth-policy.txt
path "secret/data/apigee/orgencryptionkeys" {
  capabilities = ["read"]
}
cat envencryptionkeys-auth-policy.txt
path "secret/data/apigee/envencryptionkeys" {
  capabilities = ["read"]
}

vault policy write apigee-axhashsalt-auth axhashsalt-auth-policy.txt
vault policy write apigee-redis-auth redis-auth-policy.txt
vault policy write apigee-orgencryptionkeys-auth orgencryptionkeys-auth-policy.txt
vault policy write apigee-envencryptionkeys-auth envencryptionkeys-auth-policy.txt

Create a script called generate-encoded-sas.sh with the following contents:

# generate-encoded-sas.sh

ORG=$APIGEE_ORG            # Apigee organization name
ENVS=$APIGEE_ENV_LIST      # comma separated env names, for example: dev,prod

ORG_SHORT_NAME=$(echo $ORG | head -c 15)
ENCODE=$(echo -n $ORG | shasum -a 256 | head -c 7)
ORG_ENCODE=$(echo "$ORG_SHORT_NAME-$ENCODE")
NAMES="apigee-manager,apigee-redis-default,apigee-redis-envoy-default,apigee-mart-${ORG_ENCODE},apigee-mint-task-scheduler-${ORG_ENCODE}"

for ENV in ${ENVS//,/ }
do
  ENV_SHORT_NAME=$(echo $ENV | head -c 15)
  ENCODE=$(echo -n $ORG:$ENV | shasum -a 256 | head -c 7)
  ENV_ENCODE=$(echo "$ORG_SHORT_NAME-$ENV_SHORT_NAME-$ENCODE")
  NAMES+=,apigee-runtime-${ENV_ENCODE},apigee-synchronizer-${ENV_ENCODE}
done

echo $NAMES

Run the script to generate the service account name list to bind the policies to:
```
chmod +x ./generate-encoded-sas.sh
./generate-encoded-sas.sh
```
The output should list the encoded service account names.

Using the policies, create Vault roles which bind the required Apigee service accounts.

vault write auth/kubernetes/role/apigee-axhashsalt \
	bound_service_account_names=BOUND_SA_NAMES \
	bound_service_account_namespaces=APIGEE_NAMESPACE \
	policies=apigee-axhashsalt-auth \
	ttl=1m

vault write auth/kubernetes/role/apigee-redis \
	bound_service_account_names=BOUND_SA_NAMES \
	bound_service_account_namespaces=APIGEE_NAMESPACE \
	policies=apigee-redis-auth \
	ttl=1m

vault write auth/kubernetes/role/apigee-orgencryptionkeys \
	bound_service_account_names=BOUND_SA_NAMES \
	bound_service_account_namespaces=APIGEE_NAMESPACE \
	policies=apigee-orgencryptionkeys-auth \
	ttl=1m

vault write auth/kubernetes/role/apigee-envencryptionkeys \
	bound_service_account_names=BOUND_SA_NAMES \
	bound_service_account_namespaces=APIGEE_NAMESPACE \
	policies=apigee-envencryptionkeys-auth \
	ttl=1m

Create SecretProviderClass objects.

Add the following secrets via the SecretProviderClass resources. These resources tell the CSI driver what provider to communicate with when requesting secrets. The following table shows the file names (objectNames) expected by Apigee hybrid:

Secret	Expected secret file names
AX Hash Salt	`ax-hash-salt`
Redis	`redis-password`
Org Encryption Keys	`kmsEncryptionKey` `kvmEncryptionKey` `contractEncryptionKey`
Env Encryption Keys	`kmsEncryptionKey` `kvmEncryptionKey` `envKvmEncryptionKey` `cacheEncryptionKey`

Use the following SecretProviderClass templates to configure these resources:

# axhashsalt-spc.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: apigee-axhashsalt-spc
spec:
  provider: vault
  parameters:
    roleName: apigee-axhashsalt
    vaultAddress: VAULT_ADDRESS
    # "objectName" is an alias used within the SecretProviderClass to reference
    # that specific secret. This will also be the filename containing the secret.
    # Apigee Hybrid expects these exact values so they must not be changed.
    # "secretPath" is the path in Vault where the secret should be retrieved.
    # "secretKey" is the key within the Vault secret response to extract a value from.
    objects: |
      - objectName: "ax-hash-salt"
        secretPath: ""
        secretKey: ""

# redis-spc.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: apigee-redis-spc
spec:
  provider: vault
  parameters:
    roleName: apigee-redis
    vaultAddress: VAULT_ADDRESS
    # "objectName" is an alias used within the SecretProviderClass to reference
    # that specific secret. This will also be the filename containing the secret.
    # Apigee Hybrid expects these exact values so they must not be changed.
    # "secretPath" is the path in Vault where the secret should be retrieved.
    # "secretKey" is the key within the Vault secret response to extract a value from.
    objects: |
      - objectName: "redis-password"
        secretPath: ""
        secretKey: ""

# orgencryptionkeys-spc.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: apigee-orgencryptionkeys-spc
spec:
  provider: vault
  parameters:
    roleName: apigee-orgencryptionkeys
    vaultAddress: VAULT_ADDRESS
    # "objectName" is an alias used within the SecretProviderClass to reference
    # that specific secret. This will also be the filename containing the secret.
    # Apigee Hybrid expects these exact values so they must not be changed.
    # "secretPath" is the path in Vault where the secret should be retrieved.
    # "secretKey" is the key within the Vault secret response to extract a value from.
    objects: |
      - objectName: "kmsEncryptionKey"
        secretPath: ""
        secretKey: ""
      - objectName: "kvmEncryptionKey"
        secretPath: ""
        secretKey: ""
      - objectName: "contractEncryptionKey"
        secretPath: ""
        secretKey: ""

# envencryptionkeys-spc.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: apigee-envencryptionkeys-spc
spec:
  provider: vault
  parameters:
    roleName: apigee-envencryptionkeys
    vaultAddress: VAULT_ADDRESS
    # "objectName" is an alias used within the SecretProviderClass to reference
    # that specific secret. This will also be the filename containing the secret.
    # Apigee Hybrid expects these exact values so they must not be changed.
    # "secretPath" is the path in Vault where the secret should be retrieved.
    # "secretKey" is the key within the Vault secret response to extract a value from.
    objects: |
      - objectName: "cacheEncryptionKey"
        secretPath: ""
        secretKey: ""
      - objectName: "kvmEncryptionKey"
        secretPath: ""
        secretKey: ""
      - objectName: "envKvmEncryptionKey"
        secretPath: ""
        secretKey: ""
      - objectName: "kmsEncryptionKey"
        secretPath: ""
        secretKey: ""

VAULT_ADDRESS is the endpoint where your Vault server is running. If Vault is running in the same cluster and namespace as Apigee, the format will generally be http://vault.APIGEE_NAMESPACE.svc.cluster.local:VAULT_SERVICE_PORT.

Apply the above SecretProviderClasses to the APIGEE_NAMESPACE namespace:

kubectl -n APIGEE_NAMESPACE apply -f axhashsalt-spc.yaml
kubectl -n APIGEE_NAMESPACE apply -f redis-spc.yaml
kubectl -n APIGEE_NAMESPACE apply -f orgencryptionkeys-spc.yaml
kubectl -n APIGEE_NAMESPACE apply -f envencryptionkeys-spc.yaml

Enable the external secret for the AX Hash Salt.
1. Within your overrides.yaml file, add the following configuration to enable external secret usage for the AX Hash Salt:
```
axHashSaltSecretProviderClass: apigee-axhashsalt-spc
```
2. Apply the change by upgrading the org Helm chart:
```
helm upgrade org apigee-org/ \
  --namespace APIGEE_NAMESPACE \
  -f overrides.yaml
```
Enable the external secret for the Redis password.
1. Within your overrides.yaml file, add the following configuration to enable external secret usage for the Redis password:
```
redis:
  auth:
    secretProviderClass: apigee-redis-spc
```
2. Then apply the changes by upgrading the operator and redis charts in the following order:
```
helm upgrade operator apigee-operator/ \
  --namespace APIGEE_NAMESPACE \
  -f overrides.yaml
helm upgrade redis apigee-redis/ \
  --namespace APIGEE_NAMESPACE \
  -f overrides.yaml
```
Enable the external secret for the encryption keys
1. Within your overrides.yaml file, add the following configuration to enable external secret usage for the org-level encryption keys:
```
encryptionKeySecretProviderClass: apigee-orgencryptionkeys-spc
```
2. Apply change by upgrading the org Helm chart:
```
helm upgrade org apigee-org/ \
  --namespace APIGEE_NAMESPACE \
  -f overrides.yaml
```
3. Within your overrides.yaml file for each environment, add the following configuration for the env-specific encryption keys:
```
envs:
- name: ENV_NAME
  encryptionKeySecretProviderClass: apigee-envencryptionkeys-spc
```
4. Apply the change by upgrading the env Helm chart once for each environment:
```
helm upgrade ENV_NAME apigee-env/ \
  --namespace APIGEE_NAMESPACE \
  --set env=ENV_NAME \
  -f overrides.yaml
```

Rollback

AX Hash Salt

Within your overrides.yaml file, remove the configuration which enabled external secret usage for the AX Hash Salt:
```
# Comment out or delete the following line:
# axHashSaltSecretProviderClass: apigee-axhashsalt-spc
```

Apply the change by upgrading the org Helm chart:

helm upgrade org apigee-org/ \
  --namespace APIGEE_NAMESPACE \
  -f overrides.yaml

Redis password

Within your overrides.yaml file, remove the configuration which enabled external secret usage for the Redis password:
```
redis:
  auth:
  # Comment out or delete the following line:
  # secretProviderClass: apigee-redis-spc
```

Then apply the changes by upgrading the redis and operator charts in the following order:

helm upgrade redis apigee-redis/ \
  --namespace APIGEE_NAMESPACE \
  -f overrides.yaml
helm upgrade operator apigee-operator/ \
  --namespace APIGEE_NAMESPACE \
  -f overrides.yaml

Encryption keys

Within your overrides.yaml file, remove the configuration which enabled external secret usage for the env encryption keys:

envs:
  - name: ENV_NAME
  # Comment out or delete the following line:
  # encryptionKeySecretProviderClass: apigee-envencryptionkeys-spc

Apply the change by upgrading the env Helm chart once for each environment:

helm upgrade ENV_NAME apigee-env/ \
  --namespace APIGEE_NAMESPACE \
  --set env=ENV_NAME \
  -f overrides.yaml

Within your overrides.yaml file, remove the configuration which enabled external secret usage for the org encryption keys:
```
# Comment out or delete the following line:
# encryptionKeySecretProviderClass: apigee-orgencryptionkeys-spc
```

And then Upgrade the org Helm chart:

helm upgrade org apigee-org/ \
  --namespace APIGEE_NAMESPACE \
  -f overrides.yaml