Redis production configurations by example

source: redislabs.com

This is the seventh Post of The Redis Series.

Part One: Install Redis inside Ubuntu VM
Part Two: Redis Persistence by Example
Part Three: Implement game leaderboard using Redis
Part Four: Implement Job Queue using Redis
Part Five: Building REST API backed by Redis
Part Six: Building Chat Service in Golang and Websockets backed by Redis
Part Seven: Redis Cluster configurations by example 👈
Part Eight: Redis Geospatial by example

We can run Redis in production in one of the following configurations:

1. High Available Redis (w/wo Sentinel)
2. Clustered Redis
3. High Available clustered Redis

We will take about each one and see by example how to configure each one of them, in the end, we will test automatic failover as part of HA clustered Redis.

We will use vermin to create a Virtual Machine that we will use for testing.

High Available Redis

This is the simplest and most straight forward configuration. It is suitable if your data can be fit inside one machine’s memory. However, it doesn’t support automatic failover (check the next section about Sentinel to see how to apply automatic failover). existing clients can be used without the need to support Redis cluster features. You need two machines, 1 master and 1 slave.

Example:

Let’s create Ubuntu VM and connect to it so we can start experimenting:

$ vermin create ubuntu/focal
⣽ Creating vm_03 from image ubuntu/focal
....
....
VM is ready, to connect to it use:
$ vermin ssh vm_03

$ vermin ssh vm_03

Now let’s install Redis on ubuntu:

vermin@verminbox:~$ sudo apt update -y && sudo apt install redis-server -y

The installation process will run redis-server for us on port 6379:

vermin@verminbox:~$ ps aux | grep [r]edis
redis       2373  0.1  0.4  50188  4800 ?        Ssl  18:20   0:00 /usr/bin/redis-server 127.0.0.1:6379

Let’s start another redis-server process on port 6380 to act as the slave:

vermin@verminbox:~$ /usr/bin/redis-server --port 6380 &
....
....
Running in standalone mode
....

Now let’s connect to the second server and make it slave to the first one:

vermin@verminbox:~$ redis-cli -p 6380
127.0.0.1:6380> slaveof localhost 6379
OK

Let’s try to write to the master and read from the slave:

master-slave replication

🎉🎉🎉

Hight Available Redis with Sentinel

Sentinel provides automatic failover to high available Redis configuration among other things like monitoring and notification.

It is important to note that, Sentinel (like Redis cluster) require different client capabilities and not all existing Redis client has support for it. for example, in go-redis documentation there are 3 different Redis client: Client, SentinelClient and ClusterClient.

Sentinel requires at least 3 nodes to work. it is used to monitor for the master node and when it becomes unavailable (based on quorum configuration) it starts to promote one of the slaves (auto-discovered automatically) as a master.

sentinel.conf is the configuration file for sentinel and looks like this for a single master node:

sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 60000
sentinel failover-timeout mymaster 180000
sentinel parallel-syncs mymaster 1

check the documentation to learn more about setup.

Clustered Redis

This configuration used if your data doesn’t fit inside one machine’s memory. By default, you will need a minimum of 3 nodes for it to work. Your data will be sharded (partitioned) across the 3 nodes using hash ranges (hash-slots).

This configuration is not high-available, meaning if a node failed, then there will no backup for it and we lose the data.

Not all existing Redis clients support the Clustered Redis, you need to check the documentation of the client.

Redis cluster instance requires two open ports, the default one used in standalone mode and it used RESP which is a text protocol, and another port for cluster configuration and management which is a binary protocol.

It is important to note that, Redis doesn't support transactions on keys distributed on different shards (nodes). Hash tags used to make different keys target the same shard and make transaction in clusterd mode applicable.

Example:

We will create a cluster of 3 nodes (the minimum).

First, let’s create the configuration files:

vermin@verminbox:~$ for i in 0 1 2 
do
 p=$((7000 + i))
 mkdir -p node_$p 
 cat << EOF > node_$p/redis.conf
 port $p
 cluster-enabled yes
 cluster-config-file cluster.conf
 cluster-node-timeout 5000
 appendonly yes
EOF
done

make sure we have 3 files created in the current directory:

vermin@verminbox:~$ ls
node_7000  node_7001  node_7002

Let’s run 3 Redis instances using the configuration above:

vermin@verminbox:~$ for i in 0 1 2; do cd node_$((7000 + i)); /usr/bin/redis-server redis.conf& cd -; done

Let’s make sure the 3 nodes are running:

vermin@verminbox:~$ ps aux | grep "700[012]"
vermin     16150  0.1  0.6  51700  7004 pts/1    Sl   19:23   0:00 /usr/bin/redis-server *:7000 [cluster]
vermin     16151  0.1  0.7  51700  7104 pts/1    Sl   19:23   0:00 /usr/bin/redis-server *:7001 [cluster]
vermin     16152  0.0  0.7  51700  7036 pts/1    Sl   19:23   0:00 /usr/bin/redis-server *:7002 [cluster]

Now create our cluster from the 3 nodes we just started (need to type yes when asked):

vermin@verminbox:~$ redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002
>>> Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: ad39bfb31e448456756ff3c0b0aeb63a12ff563a 127.0.0.1:7000
   slots:[0-5460] (5461 slots) master
M: 4c09bf1d8caa4048706e4440ac0a87e684628a22 127.0.0.1:7001
   slots:[5461-10922] (5462 slots) master
M: 99a53583f5a04cd6fbd92f16b3fbf437860b6380 127.0.0.1:7002
   slots:[10923-16383] (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
......
Waiting for the cluster to join
...
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: ad39bfb31e448456756ff3c0b0aeb63a12ff563a 127.0.0.1:7000
   slots:[0-5460] (5461 slots) master
M: 99a53583f5a04cd6fbd92f16b3fbf437860b6380 127.0.0.1:7002
   slots:[10923-16383] (5461 slots) master
M: 4c09bf1d8caa4048706e4440ac0a87e684628a22 127.0.0.1:7001
   slots:[5461-10922] (5462 slots) master

let’s test the cluster from redis-cli :

vermin@verminbox:~$ redis-cli -c -p 7000
127.0.0.1:7000> set x 100
-> Redirected to slot [16287] located at 127.0.0.1:7002
OK
127.0.0.1:7002> get x
"100"

Note, when we set the value of x, the cluster locate it at node 7002 despite that, we are connecting to the node 7000

High Available clustered Redis

This configuration is the same as the previous one, except we add a minimum of 1 replica node for each master one for each master server. so the minimum configuration here is 6 nodes. so we now have High availability and automatic failover (will see in the example section below).

The HA clustered Redis uses similar configuration as the clustered Redis, the only difference is that we need to start 6 nodes as a minimum and when we start the cluster we use the flag --cluster-replicas 1 , let’s see an example creating command:

Example:

First, create the configurations for 6 machines:

vermin@verminbox:~$ for i in 0 1 2 3 4 5
do
 p=$((7000 + i))
 mkdir -p node_$p 
 cat << EOF > node_$p/redis.conf
 port $p
 cluster-enabled yes
 cluster-config-file cluster.conf
 cluster-node-timeout 5000
 appendonly yes
EOF
done

Second, start the 6 machines:

vermin@verminbox:~$ for i in 0 1 2 3 4 5; do cd node_$((7000 + i)); /usr/bin/redis-server redis.conf& cd -; done

Third, let’s start the cluster on the 6 machines (need to type yes when asked):

vermin@verminbox:~$ redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 \
127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 \
--cluster-replicas 1

This command creates a cluster of 3 master and 3 replicas.

Testing automatic failover

First, let’s check our cluster using CLUSTER NODES:

vermin@verminbox:~$ redis-cli -c -p 7000
127.0.0.1:7000> CLUSTER NODES
39a53d804303f855c9df28c0c286cb95d26b8da4 127.0.0.1:7004@17004 slave 2bc439369730811542be11864e7ae0928ff681d9 0 1590523800617 5 connected
f90e8ecf684407d1b7902d4de5b524e7766dbeac 127.0.0.1:7005@17005 slave 4067752947767d3a41f6362158bf37c0edb59f21 0 1590523799000 6 connected
2bc439369730811542be11864e7ae0928ff681d9 127.0.0.1:7001@17001 master - 0 1590523800517 2 connected 5461-10922
42ccd0590dd8afd2fcf0047208305d0d44772d98 127.0.0.1:7000@17000 myself,master - 0 1590523798000 1 connected 0-5460
4067752947767d3a41f6362158bf37c0edb59f21 127.0.0.1:7002@17002 master - 0 1590523799611 3 connected 10923-16383
2fdd597c4e4d9a6c6be2351a2a06535d7445ef10 127.0.0.1:7003@17003 slave 42ccd0590dd8afd2fcf0047208305d0d44772d98 0 1590523800000 4 connected

It shows that the node at port 7000,7001 and 7002 are master, so let’s kill node at 7001 it and see what will happen:

vermin@verminbox:~$ kill -9 $(ps aux | grep [7]001 | cut -d " " -f6 )

We will notice that the following will be printed:

17083:S 26 May 2020 20:11:42.211 # Connection with master lost.
17083:S 26 May 2020 20:11:42.211 * Caching the disconnected master state.
17083:S 26 May 2020 20:11:42.483 * Connecting to MASTER 127.0.0.1:7001
17083:S 26 May 2020 20:11:42.484 * MASTER <-> REPLICA sync started
17083:S 26 May 2020 20:11:42.484 # Error condition on socket for SYNC: Connection refused
......
......
17082:S 26 May 2020 20:11:47.522 * Marking node 2bc439369730811542be11864e7ae0928ff681d9 as failing (quorum reached).
.......
17083:S 26 May 2020 20:11:47.930 # Cluster state changed: fail
17083:S 26 May 2020 20:11:47.931 # Start of election delayed for 737 milliseconds (rank #0, offset 154).
.........
17081:M 26 May 2020 20:11:48.741 # Failover auth granted to 39a53d804303f855c9df28c0c286cb95d26b8da4 for epoch 7
17079:M 26 May 2020 20:11:48.741 # Failover auth granted to 39a53d804303f855c9df28c0c286cb95d26b8da4 for epoch 7
17083:S 26 May 2020 20:11:48.752 # Failover election won: I'm the new master.
.......

Now let’s test the cluster from the client, we will notice the 3 masters are still running, which means the :

redis-cli -c -p 7000
127.0.0.1:7000> set 1 a
-> Redirected to slot [9842] located at 127.0.0.1:7004
OK
127.0.0.1:7004> set 4 b
-> Redirected to slot [14039] located at 127.0.0.1:7002
OK
127.0.0.1:7002>

In the end, I need to mention that, Redis HA cluster doesn’t provide strong consistency as replication done asynchronously.

Also, I need to mention the Redis Cluster uses a concept similar to the Raft algorithm “term”.

References:

Redis cluster tutorial - Redis

CLUSTER NODES

Redis Sentinel Documentation - Redis