A practical record of troubleshooting a surge in Redis connections in Docker

On Saturday, the redis server on the production server was unavailable, and the error message was:

The status is unavailable and cannot be used until the background checker is restored. Unexpected end of stream; expected type 'Status'

As shown in the figure below, 6300 is the port where our redis server runs.

This was the first time I encountered such a problem. I thought maybe redis was down, so I used telnet ip+port. I found that it was running normally, and then I thought about entering redis to see the current connection status. At first glance, I found that there were as many as 1,903 items.

Then I thought it was caused by too many redis connections created by the code, so I checked the code.

It is found that redis is created in only one place, and it is also executed when the service is registered. That is, it is executed only once when the application starts. Then the entire project was searched and no other place called redis initialization.

I am not satisfied. Does it mean that a connection will be created every time I read and write data in redis? Does it have something to do with frequent reading and writing? I always feel that it won't work, so I create a test code to test it.

A redis environment is built locally. Before testing, check the number of connections. Currently, there is only one, which is the current cmd connection client. This is normal.

Start the test and run the program. The code creates a connection object and tests a total of 1000 writes and 1000 reads.

No matter how I test the connection, there are always 6 connections, which means that our program creates a maximum of 5 connections, of course, mainly the thread pool.

So there is definitely no problem with the basic storage and reading of this code.

But we didn’t give up on troubleshooting the code completely, because the production server was running about 6 applications through Docker. They are all connected to the same redis. Could it be caused by other applications?

Then I want to directly query the corresponding process information through any port in the redis connection list to know which applications they are.

In Linux, process information is displayed by querying the network port number.

netstat -atunlp | grep 60852 

First look at the IP corresponding to this port, for example, the first one here is 172.17.0.1. Students who are familiar with Docker should know that this IP is the Docker gateway IP. The programs in our container all communicate with our host machine through this gateway IP. We can find the gateway IP of docker through ifconfig. The second one 172.17.0.3:6379 is the container IP of redis.

At this point, we can't find the program in which container to establish a connection with.

The dumbest way is to enter the containers one by one. That is, docker exec –it test /bin/bash and then check the network connection status of the current container. This is very cumbersome and requires installing many components to execute a series of commands.

Another way is to use the lsof command, which needs to be installed if it is not available. We can find all network connections through the process.

For example, we just found that our main process is docker, and its pid is 582251.

lsof -i |grep 582251 or lsof -i -p 582251

The result is as shown in the figure below. The specific IP actually appears on the right. This IP is the specific IP address of the docker container.

Now that we know all the IPs and ports, we can download the command execution results.

First find the IP corresponding to each of your containers.

docker inspect name |grep IPAddress //name container name or id 

After finding each IP, we will collect statistics based on all the network connection information just downloaded to see which IP has the most connections. The one with the most connections must have a problem.

Then I found the program deployed in the container corresponding to this IP, and then looked at the redis configuration. It was found that the thread pool was set to 200.

In addition, I found through github that CSRedisCore also has a preheating mechanism, namely preheat, and its default value is 5 preheating connections.

Our thread pool is set to 200 plus it has a preheating mechanism for 5 connections. I don't know if 200*5=1000 will be created. I'll study the source code carefully when I have time. This is just speculation at the moment.

I have now modified redis to poolsize=5, preheat=false. The thread pool has 5 threads and the preheating mechanism is turned off.

After modifying our connection configuration and restarting the application server and redis server (in order to completely clear the established connections), we found that the number of connections decreased, but not by much. Later, it was found that the idle time of redis was too long, causing the connection pool to maintain too many connections and not be released.

We set the timeout to 30s

Execute CONFIG SET timeout 30 (the unit is seconds. This method is only a temporary modification and is valid for the current operation. Remember to modify the redis configuration file for long-term effect)

Then look at the number of connections, which will be reduced a lot at once.

Summarize:

1. When redis connections increase dramatically, first look for problems in your own application. For example, I found that the connection pool was too large, plus the default preheating mechanism. Also, try to check whether the code level is triggered multiple times when creating the connection. If so, it must be corrected. Now instances are created by injection, depending on whether the place is called multiple times.

2. Modify the redis server configuration, such as the connection idle timeout. You can also check the maximum number of connections and the default value.

This is the end of this article about troubleshooting the problem of sudden increase in Redis connections in Docker. For more relevant content on troubleshooting the problem of sudden increase in Redis connections in Docker, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope everyone will support 123WORDPRESS.COM in the future!