Deploy Redis to GKE using Spotahome

Set up your environment

In this tutorial, you use Cloud Shell to manage resources hosted on Google Cloud. Cloud Shell comes preinstalled with the software you'll need for this tutorial, including kubectl, the gcloud CLI, Helm, and Terraform.

To set up your environment with Cloud Shell, follow these steps:

1. Launch a Cloud Shell session from the Google Cloud console, by clicking Activate Cloud Shell in the Google Cloud console. This launches a session in the bottom pane of the Google Cloud console.

2. Set environment variables:

   export PROJECT_ID=PROJECT_ID
   export KUBERNETES_CLUSTER_PREFIX=redis
   export REGION=us-central1
   

   Replace PROJECT_ID with your Google Cloud project ID.

3. Clone the GitHub repository:

   git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples
   

4. Change to the working directory:

   cd kubernetes-engine-samples/databases/redis-spotahome
   

Create your cluster infrastructure

In this section, you run a Terraform script to create a private, highly-available, regional GKE cluster. The following steps allow public access to the control plane.

You can install the operator using a Standard or Autopilot cluster.

export GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token)
terraform -chdir=terraform/gke-standard init
terraform -chdir=terraform/gke-standard apply -var project_id=${PROJECT_ID} \
  -var region=${REGION} \
  -var cluster_prefix=${KUBERNETES_CLUSTER_PREFIX}

...
Apply complete! Resources: 14 added, 0 changed, 0 destroyed.
...

export GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token)
terraform -chdir=terraform/gke-autopilot init
terraform -chdir=terraform/gke-autopilot apply -var project_id=${PROJECT_ID} \
  -var region=${REGION} \
  -var cluster_prefix=${KUBERNETES_CLUSTER_PREFIX}

...
Apply complete! Resources: 12 added, 0 changed, 0 destroyed.
...

Connect to the cluster

Using Cloud Shell, configure kubectl to communicate with the cluster:

gcloud container clusters get-credentials ${KUBERNETES_CLUSTER_PREFIX}-cluster --region ${REGION}

Deploy the Spotahome operator to your cluster

In this section, you deploy the Spotahome operator to your Kubernetes cluster using a Helm chart and then deploy a Redis cluster.

1. Add the Spotahome Redis operator Helm Chart repository:

   helm repo add redis-operator https://spotahome.github.io/redis-operator
   

2. Add a namespace for the Spotahome operator and the Redis cluster:

   kubectl create ns redis
   

3. Deploy the Spotahome operator using the Helm command-line tool:

   helm install redis-operator redis-operator/redis-operator --version 3.2.9 -n redis
   

4. Check the deployment status of the Spotahome operator using Helm:

   helm ls -n redis
   

   The output is similar to the following:

   NAME             NAMESPACE    REVISION    UPDATED                                STATUS      CHART                   APP VERSION
   redis-operator    redis      1           2023-09-12 13:21:48.179503 +0200 CEST    deployed    redis-operator-3.2.9    1.2.4
   

Deploy Redis

The basic configuration for the Redis cluster instance includes the following components:

• Three replicas of Redis nodes: one leader and two read replicas.
• Three replicas of Sentinel nodes, forming a quorum.
• CPU resource allocation of one CPU request and two CPU limits, with 4 GB memory requests and limits for Redis, and 100 m/500 m CPU and 500 MB for Sentinel.
• Tolerations, nodeAffinities, and topologySpreadConstraints configured for each workload, ensuring proper distribution across Kubernetes nodes, utilizing their respective node pools and different availability zones.

This configuration represents the minimal setup required to create a production-ready Redis cluster.

Create a basic Redis cluster

1. Create a Secret with user credentials:

   export PASSWORD=$(openssl rand -base64 12)
   kubectl create secret generic my-user -n redis \
       --from-literal=password="$PASSWORD"
   

   The operator doesn't have a feature to generate credentials, and the database password should be pre-generated.

2. Create a new Redis cluster using the basic configuration:

   kubectl apply -n redis -f manifests/01-basic-cluster/my-cluster.yaml
   

   This command creates a RedisFailover custom resource of the Spotahome operator that specifies CPU, memory requests, and limits; and taints and affinities to distribute the provisioned Pod replicas across Kubernetes nodes.

3. Wait a few minutes while Kubernetes starts the required workloads:

   kubectl wait pods -l redisfailovers.databases.spotahome.com/name=my-cluster --for condition=Ready --timeout=300s -n redis
   

4. Verify that the Redis workloads were created:

   kubectl get pod,svc,statefulset,deploy,pdb -n redis
   

   The output is similar to the following:

   NAME                                READY   STATUS  RESTARTS   AGE
   pod/redis-operator-5dc65cb7cc-krlcs   1/1   Running   0         49m
   pod/rfr-my-cluster-0                2/2     Running   0         60s
   pod/rfr-my-cluster-1                2/2     Running   0         60s
   pod/rfr-my-cluster-2                2/2     Running   0         60s
   pod/rfs-my-cluster-8475dfd96c-h5zvw   1/1   Running   0         60s
   pod/rfs-my-cluster-8475dfd96c-rmh6f   1/1   Running   0         60s
   pod/rfs-my-cluster-8475dfd96c-shzxh   1/1   Running   0         60s
   
   NAME                    TYPE        CLUSTER-IP  EXTERNAL-IP   PORT(S)   AGE
   service/redis-my-cluster ClusterIP   10.52.14.87   <none>       6389/TCP    55s
   service/redis-operator   ClusterIP   10.52.13.217   <none>      9710/TCP    49m
   service/rfr-my-cluster   ClusterIP   None           <none>      9121/TCP    61s
   service/rfs-my-cluster   ClusterIP   10.52.15.197   <none>      26379/TCP   61s
   
   NAME                            READY   AGE
   statefulset.apps/rfr-my-cluster   3/3   61s
   
   NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/redis-operator   1/1    1           1           50m
   deployment.apps/rfs-my-cluster   3/3    3           3           62s
   
   NAME                                        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
   poddisruptionbudget.policy/rfr-my-cluster   2               N/A             1                   64s
   poddisruptionbudget.policy/rfs-my-cluster   2               N/A             1                   63s
   

The operator creates the following resources:

• A Redis StatefulSet and Sentinel Deployment
• Three Pod replicas for Redis
• Three Pod replicas for Sentinel
• Two PodDisruptionBudgets, ensuring a minimum of two available replicas for cluster consistency
• The rfr-my-cluster Service, which exposes Redis metrics
• The redis-my-cluster Service, which targets the Redis cluster leader node
• The rfs-my-cluster Service, which allows clients to connect to the cluster through the Sentinels. Sentinel support is required for client libraries.

Share Redis credentials

You can share Redis credentials with clients using the Spotahome operator legacy authentication method

You must use a database password using the requirepass setting. This password is then used by all clients. To manage additional users, use Redis CLI commands.

apiVersion: databases.spotahome.com/v1
kind: RedisFailover
metadata:
  name: my-cluster
spec:
  ...
  auth:
    secretPath: my-user

Sharing Redis credentials with clients using this method has the following limitations:

• Spotahome does not provide Custom Resources for user management. You can store credentials in Secrets and referred to them in auth specs.
• There is no method to secure the connections with the TLS encryption using the custom resource.
• Live updating of credentials is not supported.

Connect to Redis

You can deploy a Redis client and authenticate using a password stored in a Kubernetes Secret.

1. Run the client Pod to interact with your Redis cluster:

   kubectl apply -n redis -f manifests/02-auth/client-pod.yaml
   

   The PASS environment variable takes the my-user Secret from the vault.

2. Wait for the Pod to be ready, then connect to it:

   kubectl wait pod redis-client --for=condition=Ready --timeout=300s -n redis
   kubectl exec -it redis-client -n redis -- /bin/bash
   

3. Verify that the connection works:

   redis-cli -h redis-my-cluster -a $PASS --no-auth-warning SET my-key "testvalue"
   

   The output is similar to the following:

   OK
   

4. Get the my-key value:

   redis-cli -h redis-my-cluster -a $PASS --no-auth-warning GET my-key
   

   The output is similar to the following:

   "testvalue"
   

5. Exit the Pod shell

   exit
   

Understand how Prometheus collects metrics for your Redis cluster

The following diagram shows how Prometheus metrics collecting works:

In the diagram, a GKE private cluster contains:

• A Redis Pod that gathers metrics on path / and port 9121
• Prometheus-based collectors that process the metrics from the Redis Pod
• A PodMonitoring resource that sends metrics to Cloud Monitoring

Google Cloud Managed Service for Prometheus supports metrics collection in the Prometheus format. Cloud Monitoring uses an integrated dashboard for Redis metrics.

The Spotahome operator exposes cluster metrics in Prometheus format using the redis_exporter as a sidecar.

1. Create the PodMonitoring resource to scrape metrics by labelSelector:

   kubectl apply -n redis -f manifests/03-prometheus-metrics/pod-monitoring.yaml
   

2. In the Google Cloud console, go to the GKE Clusters Dashboard page.

   Go to GKE Clusters Dashboard

   The dashboard shows non-zero metrics ingestion rate.

3. In the Google Cloud console, go to the Dashboards page.

   Go to Dashboards

4. Open the Redis Prometheus Overview dashboard. The dashboard shows the amount of connections and keys. It might take several minutes for the dashboard to auto-provision.

5. Connect to the client Pod and prepare variables:

   kubectl exec -it redis-client -n redis -- /bin/bash
   

6. Use the redis-cli tool to create new keys:

   for i in {1..50}; do \
     redis-cli -h redis-my-cluster -a $PASS \
     --no-auth-warning SET mykey-$i "myvalue-$i"; \
   done
   

7. Refresh the page and observe that the Commands Per Second and Keys graphs have been updated to show the actual database state.

8. Exit the Pod shell

   exit