Here I will show you how to quickly setup a Redis master-slave topology in k8s using Terraform.
If you just want to quickly get the tf file and folder structure, you can find it here: https://github.com/wifiwolfg/redis-k8s-terraform
Key points to consider
If you want to test this locally, you will need:
- A local working cluster.
Versions I used for this post:
- Minikube v1.25.2
- Kubernetes v1.23
- Terraform v1.2.2
- TF kubernetes provider v2.11.0
Before you start
The approach I am taking here is with internal TF resource references. This avoids certain human errors like mistyping the namespace, for example. The complete configuration can be inside one terraform file but here I will explain it part by part.
If you want to go directly to the terraform and redis resources, visit the repo here: https://github.com/wifiwolfg/redis-k8s-terraform
This is very straight forward. The following block creates a namespace called redis.
The headless service
Why do we need a headless service?
In simple terms, because it is a requirement when you use StatefulSets. StatefulSets are ideal when you are looking for “quoting the k8s docs”:
- Stable, unique network identifiers.
- Stable, persistent storage.
- Ordered, graceful deployment and scaling.
- Ordered, automated rolling updates.
Deploying the headless service
When setting a cluster_ip = “None”, you are creating a headless service. The namespace makes reference to the previous resource we created. As you see, we don’t need to actually type the namespace name, we just reference the resource name.
There is a nice tip on the following block and is, in fact, you can reference a complete configmap from a different file, you don’t need to actually write it all inside the configuration. I put two different examples here, one with a hardcoded configuration “slave.conf”, and one with the reference “master.conf”.
This will look for a configmap folder in your root directory that should contain the master.conf file. You can also verify it here: https://github.com/wifiwolfg/redis-k8s-terraform
There are several things going on here so I will break it down a bit. First, let’s take a look at the whole configuration:
It assigns an ordinal number to the pod’s name. In this case, the master will always be the one with 0, which will get assigned the master.conf file, and the slave will always be the ones with 1+, which would get assigned the slave.conf file.
On the volume_mount section ,inside the init container, we have “config-map” with a /mnt/ mount path, which later, on the volumes section, puts the configmap resource in that path. This is not a PV. Also we are doing a Terraform reference here.
The other volume mount is the redis-claim with an /etc mount path, which is where the init container will put the redis-server configuration files. This is a PV.
Last but not least, we have a volume mount for the redis-data with a /data mount path, which is the Redis working directory specified in this line of the redis configuration.
At the end of the configuration, we have two volume claims, one for the redis-data, and one for the redis-claim with the storage size and the access mode specified. This should dynamically create the persistent volumes if your cluster is able to do so. You can find more information about that here: https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/#using-dynamic-provisioning
Run your Terraform plan and apply it. You should see the namespace, service, statefulset, and configmap resources created. Also the pod should be up and running:
Let’s scale it up to 2 replicas:
kubectl scale -n redis statefulset redis-ss --replicas=2
We will see the slave replica running with an ordinal number of 1.
On the replica configmap, we hardcoded the slave.conf. One of the lines was slaveof redis-ss-0.redis-service.redis 6379, which means it will automatically start replicating from the master. Let’s confirm that by checking the replica logs:
kubectl logs -n redis redis-ss-1
It looks like is working perfectly!
Thanks for reading. If you have any questions, please feel free to drop a comment.