How to deploy Redis 6.x cluster through Docker

System environment:

• Redis version: 6.0.8
• Docker version: 19.03.12
• System version: CoreOS 7.8
• Kernel version: 5.8.5-1.el7.elrepo.x86_64

1. What is Redis Cluster Mode

The Redis cluster mode was officially launched after the Redis 3.0 version. This mode is a distributed solution for Redis. It is a set of programs that provide data sharing among multiple Redis nodes. The Redis cluster is decentralized. Each of its Master nodes can read and write data. Each node has an equal relationship, and each node maintains its own data and the status of the entire cluster.

The main purpose of Redis cluster design is to enable Redis data storage to scale linearly and provide a certain degree of availability through partitioning. In the actual environment, it can continue to process commands when a node goes down or is unreachable. However, if a major failure occurs (for example, when most of the master stations are unavailable), the cluster will stop running, that is, the redis cluster cannot guarantee strong consistency of data.

2. Why do we need Redis cluster?

• High availability: In Redis cluster mode, each Master is in master-slave replication mode. The data on the Master node will be synchronized to the Slave node in real time. When the Master node is unavailable, the corresponding Slave node identity will be changed to the Master node to ensure the availability of the cluster.
• Horizontal expansion of data: Redis is an in-memory database. All data is stored in memory. The memory that the server can provide in a single node is limited. When the amount of data reaches a certain level, the memory will not be enough to support the storage of so much data. At this time, the data needs to be stored in shards. The Redis cluster mode stores data in shards, which is very convenient for horizontal expansion.

3. Data Sharding of Redis Cluster

Redis cluster does not use consistent hashing, but introduces the concept of "hash slot". The Redis cluster has 16384 hash slots. Each key is checked by CRC16 and then modulo 16384 to determine which slot to place it in. Each node in the cluster is responsible for a part of the hash slots.

For example, if the current cluster has 3 nodes, then:

• Node A contains hash slots 0 to 5460;
• Node B contains hash slots 5461 to 10922;
• Node C contains hash slots 10923 to 16383;

This structure makes it easy to "add" or "delete" nodes. For example, if I want to add a new node D, I need to transfer some slots from nodes A, B, and C to node D. If I want to remove node A, I need to move the slots in A to nodes B and C, and then remove the A node without any slots from the cluster. Since moving a hash slot from one node to another does not stop the service, adding, deleting, or changing the number of hash slots on a node will not cause the cluster to become unavailable.

Redis supports multiple key operations, as long as these keys are executed in a single command (or a transaction, or Lua script execution), they belong to the same Hash slot. You can also use the hash tags command to force multiple keys to be in the same hash slot.

4. Master-slave replication of Redis cluster

Master-slave model in a cluster

In Redis cluster mode, in order to prevent some nodes in the cluster from becoming unavailable due to downtime, the Redis cluster uses the master-slave replication mode. In this mode, the Redis cluster must have at least six nodes, three of which are master nodes that can provide external reading and writing. There are also three nodes that are slave nodes, which will synchronize the data of their corresponding master nodes. When a master node has a problem and becomes unavailable, Redis will select a node from the slave nodes corresponding to the master node through an election algorithm (if there are multiple slave nodes for the master node), change it to a new master node, and enable it to provide services to the outside world.

For example, there are three master nodes A, B, and C and their corresponding six slave nodes (A1, A2), (B1, B2), and (C1, C2), a total of nine nodes. If node A fails, then its corresponding slave nodes A1 and A2 will use the election algorithm to select one of the nodes to be promoted to the master node to ensure that the cluster can provide normal services. However, when the two slave nodes A1 and A2 or more than half of the master nodes are unavailable, the cluster is also unavailable.

When deploying Redis cluster mode, at least six nodes are required to form a cluster to ensure the availability of the cluster.

Related concepts of master-slave replication

(1) Full replication and incremental replication

In Redis master-slave replication, there are two data synchronization methods: "full replication" and "incremental replication":

• Full replication: used for initial replication or other situations where partial replication is not possible, all data in the master node is sent to the slave node. When the amount of data is too large, it will cause a lot of network overhead.
• Incremental replication: It is used to handle data loss scenarios caused by network crashes and other reasons in master-slave replication. When the slave node connects to the master node again, if conditions permit, the master node will resend the lost data to the slave node. Because the resent data is much smaller than the full data, it can effectively avoid the excessive overhead of full replication. However, please note that if the network interruption time is too long and the master node is unable to completely save the write commands executed during the interruption, partial replication cannot be performed and full replication will still be used.

(2) Record the offset of the copy position

• The master and slave nodes participating in replication will maintain their own replication offsets. After processing the write command operation, the master node will record the byte length of the command cumulatively. You can use the info replication command to query master_repl_offset offset information.
• The slave node reports its own replication offset to the master node every second, so the master node also saves the replication offset of the slave node.
• After receiving the command sent by the master node, the slave node will also accumulate and record its own offset. You can use the info replication command to query slave_repl_offset offset information.

(3) Copy the backlog buffer

The replication backlog buffer is a fixed-length queue stored on the master node. The default size is 1MB. It is created when the master node has a connected slave node. When the master node responds to a write command, it not only sends the command to the slave node, but also writes it to the replication backlog buffer as a backup of the write command.

In addition to storing write commands, the replication backlog buffer also stores the replication offset corresponding to each byte in it. Since the replication backlog buffer is fixed-length and first-in-first-out, it stores the write commands most recently executed by the primary node, and older write commands will be squeezed out of the buffer.

(4) Node running ID

• After each Redis node is started, a 40-bit hexadecimal string is dynamically assigned as the running ID. The main function of the running ID is to uniquely identify the Redis node. For example, the slave node saves the running ID of the master node to identify which master node it is replicating. If only IP + Port is used to identify the master node, then the master node restarts and changes the overall data set (such as replacing RDB/AOF files). It will be unsafe for the slave node to copy data based on the offset. Therefore, when the running ID changes, the slave node will perform a full copy. You can view the running ID of the current node by using the info server command.
• It should be noted that when Redis is shut down and restarted, the running ID will change accordingly.

Execution process of master-slave replication

(1) psync command

The slave node uses psync to obtain the runid and offset from the master node.

The master node will return a response message based on its own situation. It may be FULLRESYNC runid offset to trigger full replication, or CONTINUE to trigger incremental replication.

(2) psync triggers full replication

The slave node uses the psync command to complete partial replication and full replication functions:

When the slave node connects to the master node for the first time, a full copy is performed:

① The Slave node sends the psync? - 1 command, indicating that it requires the Master to perform data synchronization;

② The master detects that there is no offset and it is the first time to perform replication. It performs full replication and sends the FULLRESYNC {runid} {offset} command to send the runid and offset to the slave node.

③ The Slave node saves the runid and offset information transmitted by the Master node.

④ The Master node executes bgsave to generate an RBD snapshot file and uses a buffer to record all commands executed from now on.

⑤ The Master node sends the RBD file to the Slave node.

⑥ The Master node sends the write command recorded in the BUFFER buffer area to the Slave node.

⑦ The Slave node clears the old data.

⑧ The Slave node loads the RBD file passed in by the Master node.

(2) psync triggers incremental replication

When the slave node is disconnected from the master node, the slave node will reconnect to the master node, which will trigger incremental replication. The process is as follows:

① The slave node uses psync to send runid and offset values.

② The Master node verifies whether the runid sent by the Slave node is the same as its own:

• Not the same: If they are not the same, perform a full copy;
• Same: You also need to verify whether the corresponding offset exists in the cache. If not, perform full copy; otherwise, perform incremental copy.

③ After the corresponding offset condition is met, verify whether the offset value in the buffer is the same as the offset sent by the Slave node:

• Same: The returned offset value is the same, and no copy operation is performed;
• Different: Send the CONTINUE offset command (Note: the offset here is the offset value recorded in the Master node cache, not the offset value passed by the Slave node);

④ The Master node copies data from the replication buffer, starting from the offset sent by the Slave node and ending with the offset recorded in the Master node buffer, and gives the data in this range to the Slave node.

5. Redis Cluster Related Concepts

Consistency in Redis Cluster

Redis mentioned in the official documentation that it cannot guarantee strong data consistency, that is, the data written to the Redis cluster may be lost under certain conditions. The main reason is that the Redis cluster uses asynchronous replication mode. The writing process is as follows:

The execution order is as follows:

① The client writes a command to any master node;

② The master node replies to the client with the status of command execution;

③ The master node passes the write operation command to its slave node;

Redis cluster strikes a balance between performance and consistency. The master node is set to send the executed command to each slave node for synchronization after receiving the client write command. This process is an asynchronous operation, and the master node will not wait for the slave node to reply but will immediately reply to the client command execution status. Reducing synchronization operations in this way can greatly improve the execution speed of the system and avoid the process of waiting for slave nodes to reply to messages becoming a bottleneck in system performance.

However, because the master node replies to the client with the command execution status without waiting for the slave node to reply after receiving the information, data loss is likely to occur when a problem occurs in the node. For example, the client sends a command to the master node, and then the master node asynchronously sends the command to its slave node for data backup, and then immediately replies to the client with the command execution status. If during this process, the master node has a problem and crashes, and the slave node also has an error when processing the synchronization data sent by the master node. If the master node happens to be down or unavailable at this time, the slave node will be converted to the new master node, and the commands executed on the previous master node will be lost.

Redis inter-cluster communication mechanism

In the Redis cluster, the data node provides two TCP ports. When configuring the firewall, you need to open the following two types of ports at the same time:

• Common port: the client access port, such as the default 6379;
• Cluster port: normal port number plus 10000, such as 6379, the cluster port is 16379, used for communication between cluster nodes;

The Gossip protocol is used for communication between nodes in the cluster. Nodes make judgments based on scheduled tasks at a fixed frequency (10 times per second). When the cluster status changes, such as adding or deleting nodes or changing slot status, the cluster status will be synchronized through communication between nodes to make the cluster converge. There are five types of Gossip messages sent between clusters:

• MEET: During the node handshake phase, a meet message is sent to the newly joined node, requesting the new node to join the current cluster. The new node will reply with a pong message after receiving the message.
• PING: Nodes send ping messages to each other, and those that receive the messages will reply with pong messages. The ping message content contains the status information of this node and other nodes, so as to achieve status synchronization;
• PONG: The pong message contains its own status data. When receiving a ping or meet message, it will reply with a pong message and will also actively broadcast a pong message to the cluster.
• FAIL: When a master node determines that another master node has entered the fail state, it broadcasts this message to the cluster. The receiving nodes save the message and make a status determination for the failed node.
• PUBLISH: When a node receives a publish command, it first executes the command and then broadcasts the publish message to the cluster. The nodes that receive the message also execute the publish command.

Redis cluster failure status

It is also impossible to guarantee 100% cluster availability in Redis cluster mode. When an unpredictable event occurs, the Redis cluster will enter a failed state, in which case the Redis cluster will not be able to provide services normally. The conditions for entering the failure state are mainly:

① If all nodes are down, the cluster will enter the fail state;

② If more than half of the master nodes are unavailable, the cluster will enter the fail state;

③ If any master node fails and the master node has no corresponding slave node or all slave nodes fail, the cluster will enter the fail state;

Redis cluster resharding mechanism

Redis cluster resharding (adding/removing nodes) mechanism:

Add a new node: allocate some slots from other nodes to the new node

Delete node: delete the node's slot and assign it to another node

But these operations need to be done manually and can be performed without stopping the server.

Disadvantages of Redis Cluster

The replication structure only supports single-layer structure, not tree structure.

It does not support multiple databases. You can only use database 0 and execute the select 0 command.

The key is the minimum granularity of data partitioning. A large key-value pair cannot be mapped to different nodes.

Key transaction support is limited. When multiple keys are distributed on different nodes, transactions cannot be used. Transactions can only be supported on the same node.

Batch key operations have limited support, such as mset and mget commands. If multiple keys are mapped in different slots, these commands cannot be used normally.

Redis Cluster Configuration Parameters

We are about to create a sample cluster deployment. Before we proceed, let's cover the configuration parameters that Redis Cluster introduces in the redis.conf file.

cluster-config-file: Sets the storage location of the Redis cluster configuration information and status. This file is generated by the Redis cluster, and we can only specify its storage location.

cluster-node-timeout: Set the communication timeout of Redis cluster nodes;

cluster-migration-barrier: The minimum number of slave nodes required by the master node. Only when this number is reached, the slave nodes will migrate when the master node fails.

cluster-enabled: Whether to enable Redis cluster mode.

yes: Enable Redis cluster;

no: do not enable cluster mode;

cluster-require-full-coverage: Sets cluster availability.

yes: indicates that when the master library responsible for a slot goes offline and there is no corresponding slave library to perform failure recovery, the cluster is unavailable. This situation is demonstrated below.

no: indicates that when the master library responsible for a slot goes offline and there is no corresponding slave library to perform failure recovery, the cluster is still available. This situation is demonstrated below.

cluster-slave-validity-factor:

0: No matter how long the slave node loses connection with the master node, the slave node will try to upgrade to the master node.

Positive number: The time obtained by cluster-node-timeout * cluster-slave-validity-factor is the maximum time that the slave node data is valid after the slave node loses connection with the master node. If this time is exceeded, the slave node will not start failover. Assuming cluster-node-timeout=5 , cluster-slave-validity-factor=10 , if the slave node loses connection with the master node for more than 50 seconds, the slave node cannot become the master node.

6. Deploy Redis Cluster with Docker

1. Redis deployment machine allocation

Here we allocate the nodes of the Redis cluster to be deployed and deploy them to different machines. The arrangement is as follows:

2. Create a data storage directory

Create a directory in advance for storing Redis configuration files and persistent data:

Execute the command to create a storage directory on the first server 192.168.2.11 :

$ mkdir -p /var/lib/redis/7000 & mkdir -p /var/lib/redis/7003

Execute the command to create a storage directory on the second server 192.168.2.12 :

$ mkdir -p /var/lib/redis/7001 & mkdir -p /var/lib/redis/7004

Execute the command to create a storage directory on the third server 192.168.2.13 :

$ mkdir -p /var/lib/redis/7002 & mkdir -p /var/lib/redis/7005

3. Create a Redis configuration file

First server 192.168.2.11 configuration file:

## 7000 port configuration file $ cat > /var/lib/redis/7000/redis.conf << EOF
port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
daemonize no
protected-mode no
pidfile /data/redis.pid
EOF
 
## 7003 port configuration file $ cat > /var/lib/redis/7003/redis.conf << EOF
port 7003
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
daemonize no
protected-mode no
pidfile /data/redis.pid
EOF

Second server 192.168.2.12 configuration file

## 7001 port configuration: redis.conf
$ cat > /var/lib/redis/7001/redis.conf << EOF
port 7001
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
daemonize no
protected-mode no
pidfile /data/redis.pid
EOF
 
## 7004 port configuration: redis-7004.conf
$ cat > /var/lib/redis/7004/redis.conf << EOF
port 7004
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
daemonize no
protected-mode no
pidfile /data/redis.pid
EOF

The third server 192.168.2.13 configuration file

## 7002 port configuration: redis-7002.conf
$ cat > /var/lib/redis/7002/redis.conf << EOF
port 7002
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
daemonize no
protected-mode no
pidfile /data/redis.pid
EOF
 
## 7005 port configuration: redis-7005.conf
$ cat > /var/lib/redis/7005/redis.conf << EOF
port 7005
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
daemonize no
protected-mode no
pidfile /data/redis.pid
EOF

4. Pull the Redis image in advance

Pull the Redis image from the three servers in advance to avoid having to pull the image during execution, which would cause slow operation.

$ docker pull redis:6.0.8

5. Run and start the Redis image

The three servers execute the Docker run command to start the Redis image. It should be noted here that Dockers on different servers cannot communicate with each other, so here we set the network mode of the started container to host mode, so that the container will not create a virtual network card, but use the host's network.

• -d: Set the container to run in the background;
• -v: specifies the mounted host storage directory;
• --name: specifies the name of the container after running;
• --cpus: specifies the number of CPUs used by the container;
• --memory: limit the amount of memory used by the container;
• --memory-swap: specifies the swap memory size, which is set to 0 here, meaning no swap memory is used;
• --net: specifies the network mode used by Docker;
• --restart: specifies the restart strategy of the container when Docker restarts;
• --privileged: Set the container to have privileges and be able to obtain the root permissions of the host machine;

The first server 192.168.2.11 executes the following command

## Run Redis image on port 7000
 $ docker run -d -v /var/lib/redis/7000:/data \
--cpus=1 --memory=2GB --memory-swap=0 \
--privileged=true \
--restart=always \
--net host \
--name redis-7000 \
redis:6.0.8 redis-server /data/redis.conf
 
## Run Redis image on port 7003
 $ docker run -d -v /var/lib/redis/7003:/data \
--cpus=1 --memory=2GB --memory-swap=0 \
--privileged=true \
--restart=always \
--net host \
--name redis-7003 \
redis:6.0.8 redis-server /data/redis.conf

The second server 192.168.2.12 executes the following command

## Run Redis image on port 7001
 $ docker run -d -v /var/lib/redis/7001:/data \
--cpus=1 --memory=2GB --memory-swap=0 \
--privileged=true \
--restart=always \
--net host \
--name redis-7001 \
redis:6.0.8 redis-server /data/redis.conf
 
## Run Redis image on port 7004
 $ docker run -d -v /var/lib/redis/7004:/data \
--cpus=1 --memory=2GB --memory-swap=0 \
--privileged=true \
--restart=always \
--net host \
--name redis-7004 \
redis:6.0.8 redis-server /data/redis.conf

The third server 192.168.2.13 executes the following command:

## Run Redis image on port 7002
 $ docker run -d -v /var/lib/redis/7002:/data \
--cpus=1 --memory=2GB --memory-swap=0 \
--privileged=true \
--restart=always \
--net host \
--name redis-7002 \
redis:6.0.8 redis-server /data/redis.conf
 
## Run Redis image on port 7005
 $ docker run -d -v /var/lib/redis/7005:/data \
--cpus=1 --memory=2GB --memory-swap=0 \
--privileged=true \
--restart=always \
--net host \
--name redis-7005 \
redis:6.0.8 redis-server /data/redis.conf

6. Create a Redis cluster

Enter any server at random, use the redis-cli tool of the Redis image to execute the create cluster command to make each Redis form a cluster. Here I enter the first server 192.168.2.11 , use the Redis end image with port 7000 , and execute the following command:

-p: specifies the port to connect to Redis;

create: create a Redis cluster;

--cluster: Use Redis cluster mode commands;

--cluster-replicas: specifies the number of replicas (slave number);

$ docker exec -it redis-7000 \
redis-cli -p 7000 --cluster create \
192.168.2.11:7000 192.168.2.12:7001 192.168.2.13:7002 \
192.168.2.11:7003 192.168.2.12:7004 192.168.2.13:7005 \
--cluster-replicas 1

Then you will see the following information:

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.2.12:7004 to 192.168.2.11:7000
Adding replica 192.168.2.13:7005 to 192.168.2.12:7001
Adding replica 192.168.2.11:7003 to 192.168.2.13:7002
M: 5e50824c55d4df42db4d2987796f0c0b468c273f 192.168.2.11:7000
   slots:[0-5460] (5461 slots) master
M: 36565e0273fd62921aa1f2d85c5f7ac98a5b9466 192.168.2.12:7001
   slots:[5461-10922] (5462 slots) master
M: 0cc1aaf960defae7332e9256dd25ee5e5c99e65f 192.168.2.13:7002
   slots:[10923-16383] (5461 slots) master
S: 42d6e3979395ba93cd1352b6d17044f6b25d9379 192.168.2.11:7003
   replicates 0cc1aaf960defae7332e9256dd25ee5e5c99e65f
S: ac5d34b57a8f73dabc60d3a56469055ec64fcde7 192.168.2.12:7004
   replicates 5e50824c55d4df42db4d2987796f0c0b468c273f
S: 470b7ff823f10a309fb07311097456210506f6d8 192.168.2.13:7005
   replicates 36565e0273fd62921aa1f2d85c5f7ac98a5b9466
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 192.168.2.11:7000)
M: 5e50824c55d4df42db4d2987796f0c0b468c273f 192.168.2.11:7000
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 470b7ff823f10a309fb07311097456210506f6d8 192.168.2.13:7005
   slots: (0 slots) slave
   replicates 36565e0273fd62921aa1f2d85c5f7ac98a5b9466
S: 42d6e3979395ba93cd1352b6d17044f6b25d9379 192.168.2.11:7003
   slots: (0 slots) slave
   replicates 0cc1aaf960defae7332e9256dd25ee5e5c99e65f
S: ac5d34b57a8f73dabc60d3a56469055ec64fcde7 192.168.2.12:7004
   slots: (0 slots) slave
   replicates 5e50824c55d4df42db4d2987796f0c0b468c273f
M: 0cc1aaf960defae7332e9256dd25ee5e5c99e65f 192.168.2.13:7002
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 36565e0273fd62921aa1f2d85c5f7ac98a5b9466 192.168.2.12:7001
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

7. View cluster information

Enter the Redis image and run the redis-cli command:

-p: specifies the endpoint to connect to Redis;

-c: Use cluster mode;

$ docker exec -it redis-7000 redis-cli -p 7000 -c

View cluster information:

> cluster info
 
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:866
cluster_stats_messages_pong_sent:854
cluster_stats_messages_sent:1720
cluster_stats_messages_ping_received:849
cluster_stats_messages_pong_received:866
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1720

View cluster node information:

> cluster nodes
 
470b7ff823f10a309fb07311097456210506f6d8 192.168.2.13:7005@17005 slave 36565e0273fd62921aa1f2d85c5f7ac98a5b9466 0 1600267217000 2 connected
42d6e3979395ba93cd1352b6d17044f6b25d9379 192.168.2.11:7003@17003 slave 0cc1aaf960defae7332e9256dd25ee5e5c99e65f 0 1600267218171 3 connected
ac5d34b57a8f73dabc60d3a56469055ec64fcde7 192.168.2.12:7004@17004 slave 5e50824c55d4df42db4d2987796f0c0b468c273f 0 1600267216161 1 connected
0cc1aaf960defae7332e9256dd25ee5e5c99e65f 192.168.2.13:7002@17002 master - 0 1600267218070 3 connected 10923-16383
36565e0273fd62921aa1f2d85c5f7ac98a5b9466 192.168.2.12:7001@17001 master - 0 1600267217163 2 connected 5461-10922
5e50824c55d4df42db4d2987796f0c0b468c273f 192.168.2.11:7000@17000 myself,master - 0 1600267217000 1 connected 0-5460

Reference address:

http://www.redis.cn/topics/cluster-tutorial.html

http://www.redis.cn/topics/cluster-spec.html

https://jasonkayzk.github.io

https://www.cnblogs.com/kevingrace/p/5685332.html

This is the end of this article about how to deploy Redis 6.x cluster through Docker. For more information about deploying Redis 6.x cluster with Docker, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope you will support 123WORDPRESS.COM in the future!