Deploy Redis to GKE using Redis Enterprise

Set up your environment

In this tutorial, you use Cloud Shell to manage resources hosted on Google Cloud. Cloud Shell comes preinstalled with the software you need for this tutorial, including kubectl, the gcloud CLI, and Terraform.

To set up your environment with Cloud Shell, follow these steps:

1. Launch a Cloud Shell session from the Google Cloud console, by clicking Activate Cloud Shell in the Google Cloud console. This launches a session in the bottom pane of the Google Cloud console.

2. Set environment variables:

   export PROJECT_ID=PROJECT_ID
   export KUBERNETES_CLUSTER_PREFIX=redis
   export REGION=us-central1
   

   Replace PROJECT_ID: your Google Cloud with your project ID.

3. Clone the GitHub repository:

   git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples
   

4. Change to the working directory:

   cd kubernetes-engine-samples/databases/redis-enterprise-operator
   

Create your cluster infrastructure

In this section, you run a Terraform script to create a private, highly-available, regional GKE cluster and VPC.

The following diagram shows a private regional Standard GKE cluster deployed across three different zones:

To deploy this infrastructure, run the following commands from the Cloud Shell:

  cd terraform/gke-standard
  export GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token)
  terraform init
  terraform apply -var project_id=${PROJECT_ID}   \
    -var region=${REGION}  \
    -var cluster_prefix=${KUBERNETES_CLUSTER_PREFIX}

When prompted, type yes. It might take several minutes for this command to complete and for the cluster to show a ready status.

Terraform creates the following resources:

• A VPC network and private subnet for the Kubernetes nodes
• A router to access the internet through NAT
• A private GKE cluster in the us-central1 region
• One node pool with auto-scaling enabled (One to two nodes per zone, one node per zone minimum)

The output is similar to the following:

...
Apply complete! Resources: 14 added, 0 changed, 0 destroyed.
...

Connect to the cluster

Using Cloud Shell, configure kubectl to communicate with the cluster:

gcloud container clusters get-credentials ${KUBERNETES_CLUSTER_PREFIX}-cluster --region ${REGION}

Deploy the Redis Enterprise operator to your cluster

In this section, you deploy the Redis Enterprise operator to your Kubernetes cluster.

1. Create namespaces for the REC and its applications:

   kubectl create namespace rec-ns
   kubectl create namespace application
   

2. Label the namespaces:

   kubectl label namespace rec-ns connection=redis
   kubectl label namespace application connection=redis
   

3. Get the latest version of the Redis Enterprise Operator bundle:

   VERSION=`curl --silent https://api.github.com/repos/RedisLabs/redis-enterprise-k8s-docs/releases/latest | grep tag_name | awk -F'"' '{print $4}'`
   

4. Install the Redis Enterprise operator:

   kubectl apply -n rec-ns -f https://raw.githubusercontent.com/RedisLabs/redis-enterprise-k8s-docs/$VERSION/bundle.yaml
   

   The output is similar to the following:

   role.rbac.authorization.k8s.io/redis-enterprise-operator created
   rolebinding.rbac.authorization.k8s.io/redis-enterprise-operator created
   serviceaccount/redis-enterprise-operator created
   service/admission created
   customresourcedefinition.apiextensions.k8s.io/redisenterpriseclusters.app.redislabs.com created
   customresourcedefinition.apiextensions.k8s.io/redisenterprisedatabases.app.redislabs.com created
   customresourcedefinition.apiextensions.k8s.io/redisenterpriseremoteclusters.app.redislabs.com created
   customresourcedefinition.apiextensions.k8s.io/redisenterpriseactiveactivedatabases.app.redislabs.com created
   deployment.apps/redis-enterprise-operator created
   

Deploy Redis Enterprise Cluster

1. Apply the manifest to your cluster:

   kubectl apply -n rec-ns -f manifests/01-basic-cluster/rec.yaml
   

   The command might take several minutes to complete.

2. Check the status of the REC deployment:

   kubectl get rec -n rec-ns
   

   The output is similar to the following:

   NAME      NODES   VERSION    STATE     SPEC STATUS   LICENSE STATE   SHARDS LIMIT   LICENSE EXPIRATION DATE   AGE
   gke-rec   3       7.2.4-52   Running   Valid         Valid           4              2023-09-29T20:15:32Z      4m7s
   

   The cluster is ready when STATE is RUNNING.

Optional: Configure the admission controller

You can optionally configure infrastructure for the database validation on deployment.

1. Setup the admission controller and check if the admission tls Secret is present:

   kubectl get secret admission-tls -n rec-ns
   

2. Get the certificate:

   export CERT=$(kubectl get secret admission-tls -n rec-ns -o jsonpath='{.data.cert}')
   

3. Copy the certificate into the webhook.yaml file:

   sed -i -e 's/CRT/'$CERT'/g' manifests/01-basic-cluster/webhook.yaml
   

4. Deploy the validation webhook:

   sed -i -e 's/CRT/'$CERT'/g' manifests/01-basic-cluster/webhook.yaml
   

   The admission controller validates database syntax on labeled namespaces.

5. Verify the admission controller by creating a non-functional database:

   kubectl apply -n rec-ns -f - << EOF
   apiVersion: app.redislabs.com/v1alpha1
   kind: RedisEnterpriseDatabase
   metadata:
     name: redis-enterprise-database
   spec:
     evictionPolicy: illegal
   EOF
   

   The output is similar to the following:

   Error from server: error when creating "STDIN": admission webhook "redisenterprise.admission.redislabs" denied the request: 'illegal' is an invalid value for 'eviction_policy'. Possible values are ['volatile-lru', 'volatile-ttl', 'volatile-random', 'allkeys-lru', 'allkeys-random', 'noeviction', 'volatile-lfu', 'allkeys-lfu']
   

Create namespaces

By default, the Redis Enterprise Operator has no privileges to perform actions outside its own namespace. To allow the Redis Enterprise Operator to create REDB and database endpoints in other namespaces, you must configure RBAC.

1. Apply the corresponding role and role binding in the application namespace:

   kubectl apply -f manifests/01-basic-cluster/role.yaml -n application
   kubectl apply -f manifests/01-basic-cluster/role-binding.yaml -n application
   

2. Create cluster role and cluster role binding in the rec-ns namespace:

   kubectl apply -n rec-ns -f manifests/01-basic-cluster/cluster_role.yaml 
   kubectl apply -n rec-ns -f manifests/01-basic-cluster/cluster_role_binding.yaml
   

3. Edit the REC ConfigMap to add control over the application namespace:

   kubectl patch ConfigMap/operator-environment-config --type merge -p '{"data": {"REDB_NAMESPACES_LABEL": "connection=redis"}}' -n rec-ns
   

   Each namespace labeled as ConfigMap is patched.

4. Check the status of the resources in your Redis infrastructure in the rec-ns namespace:.

   kubectl get pod,deploy,svc,rec,statefulset,cm,secrets -n rec-ns
   

   The output is similar to the following:

   NAME                                             READY   STATUS    RESTARTS        AGE
   pod/gke-rec-0                                    2/2     Running   0               172m
   pod/gke-rec-1                                    2/2     Running   0               171m
   pod/gke-rec-2                                    2/2     Running   0               168m
   pod/gke-rec-services-rigger-5f885f59dc-gc79g     1/1     Running   0               172m
   pod/redis-enterprise-operator-6668ccd8dc-kx29z   2/2     Running   2 (5m58s ago)   5h
   
   NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/gke-rec-services-rigger     1/1     1            1           172m
   deployment.apps/redis-enterprise-operator   1/1     1            1           5h
   
   NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
   service/admission      ClusterIP   10.52.11.13   <none>        443/TCP             5h
   service/gke-rec        ClusterIP   10.52.5.44    <none>        9443/TCP,8001/TCP   172m
   service/gke-rec-prom   ClusterIP   None          <none>        8070/TCP            172m
   service/gke-rec-ui     ClusterIP   10.52.3.29    <none>        8443/TCP            172m
   
   NAME                                               NODES   VERSION    STATE     SPEC STATUS   LICENSE STATE   SHARDS LIMIT   LICENSE EXPIRATION DATE   AGE
   redisenterprisecluster.app.redislabs.com/gke-rec   3       7.2.4-52   Running   Valid         Valid           4              2023-10-05T11:07:20Z      172m
   
   NAME                       READY   AGE
   statefulset.apps/gke-rec   3/3     172m
   
   NAME                                    DATA   AGE
   configmap/gke-rec-bulletin-board        1      172m
   configmap/gke-rec-health-check          5      172m
   configmap/kube-root-ca.crt              1      5h2m
   configmap/operator-environment-config   1      5h
   
   NAME                   TYPE     DATA   AGE
   secret/admission-tls   Opaque   2      5h
   secret/gke-rec         Opaque   2      172m
   

Deploy Redis Enterprise Databases

1. Create Redis Enterprise Databases in the application namespaces:

   kubectl apply -f manifests/01-basic-cluster/a-rdb.yaml -n application
   

2. Check the REDB status:

   kubectl get redb --all-namespaces
   

   The output is similar to the following:

   NAMESPACE       NAME       VERSION   PORT    CLUSTER   SHARDS   STATUS   SPEC STATUS   AGE
   application   app-db   7.2.0     12999   gke-rec   1        active   Valid         15s
   

3. Verify that the Services for each REDB are running:

   kubectl get svc --all-namespaces
   

   The output is similar to the following:

   NAMESPACE      NAME      TYPE          CLUSTER-IP   EXTERNAL-IP                           PORT(S)    AGE
   application  app-db  ExternalName  <none>       redis-12999.rec-ns.svc.cluster.local  12999/TCP  72m
   

4. Verify that the Secret was created:

   kubectl get secrets -n application
   

   The output is similar to the following:

   NAME            TYPE     DATA   AGE
   redb-app-db   Opaque   3      96m
   

Authenticate using passwords

You can connect to REDB using a Pod with redis-cli in the application namespace. The client Pod uses the secrets available in the application namespace (REDB) to establish a connection.

Databases created with the Custom Resource REDB only support password authentication without ACL.

1. Create the client Pod:

   kubectl apply -n application -f manifests/03-auth/client_pod.yaml
   

2. Connect to the client Pod:

   kubectl exec -n application -i -t redis-client -c redis-client -- /bin/sh
   

3. Connect to the database:

   redis-cli -h $SERVICE -p $PORT --pass $PASS
   

4. Create a key:

   SET mykey "Hello World"
   

   The output is similar to the following:

   OK
   

5. Get the key:

   GET mykey
   

   The output is similar to the following:

   "Hello World"
   

6. Exit the Pod shell

   exit
   

Understand how Prometheus collects metrics for your Redis cluster

The following diagram shows how Prometheus metrics collecting works:

In the diagram, a GKE private cluster contains:

• A Redis Pod that gathers metrics on path / and port 8070
• Prometheus-based collectors that process the metrics from the Redis Pod
• A PodMonitoring resource that sends metrics to Cloud Monitoring

The Redis Enterprise operator exposes cluster metrics in Prometheus format.

1. Create the metrics-proxy Deployment:

   kubectl apply -n rec-ns -f manifests/02-prometheus-metrics/metrics-proxy.yaml
   

   Because the operator only provides an HTTPS endpoint with the self-signed certificate and the PodMonitoring resource doesn't support disabling TLS certificate verification, you use the metrics-proxy Pod as a reverse proxy for this endpoint to expose the metrics on the HTTP port.

2. Create the PodMonitoring resource to scrape metrics by labelSelector:

   kubectl apply -n rec-ns -f manifests/02-prometheus-metrics/pod-monitoring.yaml
   

3. In the Google Cloud console, go to the GKE Clusters Dashboard page.

   Go to GKE Clusters Dashboard

   The dashboard shows non-zero metrics ingestion rate.

Create a Dashboard

You can view the metrics by creating a dashboard.

1. Create the dashboard:

   gcloud --project "${PROJECT_ID}" monitoring dashboards create --config-from-file monitoring/dashboard.json
   

   The output is similar to the following:

   Created [f4efbe4e-2605-46b4-9910-54b13d29b3be].
   

2. In the Google Cloud console, go to the Dashboards page.

   Go to Dashboards

3. Open the Redis Enterprise Cluster dashboard. It might take several minutes for the dashboard to auto-provision.

Verify the exported metrics

To verify the metrics, create new database and examine the metrics.

1. Open the Redis Enterprise Cluster dashboard.

2. Create an additional Redis database:

   kubectl apply -n rec-ns -f manifests/02-prometheus-metrics/c-rdb.yaml
   

   The Database Count on the dashboard should update.

3. Create a client Pod to connect to the new database:

   kubectl apply -n rec-ns -f manifests/02-prometheus-metrics/client_pod.yaml
   

4. Connect to the client Pod and prepare variables:

   kubectl exec -it redis-client-c -n rec-ns -- /bin/bash
   

5. Use the redis-cli tool to create new keys:

   for i in {1..50}; do \
     redis-cli -h $SERVICE -p $PORT -a $PASS \
     --no-auth-warning SET mykey-$i "myvalue-$i"; \
   done
   

6. Refresh the page and observe that graphs have been updated to show the actual database state.

7. Exit the Pod shell

   exit