Introduction to the pitfalls of Linux high concurrency and performance optimization

Preface

The Linux operating system is the preferred operating system for servers today. Under the default system parameters of Linux, Linux does not provide very good support for high concurrency. The editor has been engaged in application development under Linux for many years. Regarding high concurrency under Linux systems, the editor has stepped on the pitfalls and how to solve the pitfalls. Here are a few points for your reference to avoid falling into the pits again.

Analysis and solution of Too many open files problem during Linux application running

The reason for this prompt is that the number of file socket connections opened by the program exceeds the system setting value.

Check the maximum number of files each user is allowed to open

ulimit -a

The open files (-n) 1024 indicates that the maximum number of files each user is allowed to open is 1024.

The maximum number of file handles in the current system. This is only used for viewing and cannot be set or modified.

cat /proc/sys/fs/file-max

View the open file limit for a process

cat /proc/10446(pid)/limits

Set the open files numerical method

ulimit -n 65535

This setting method will be restored to the default value after restart.

Permanent setting method:

vim /etc/security/limits.conf

Add at the end

* soft nofile 65535

* hard nofile 65535

The system needs to be restarted to take effect

After this modification, the problem was effectively solved.

Analysis and solution of excessive time_wait issues under high concurrency in Linux

The phenomenon is that in high-concurrency scenarios, the server runs applications lagging.

Troubleshooting method: Check the server configuration:

netstat -ant|awk '/^tcp/ {++S[$NF]} END {for(a in S) print (a,S[a])}'

It is found that there are too many time_wait entries, tens of thousands, which should be a large number of sockets in TIME_WAIT state. If the concurrent usage of clients continues to be high, some clients will show that they cannot connect.
TCP connection status description:

CLOSED: No connection is active or in progress LISTEN: The server is waiting for an incoming call SYN_RECV: A connection request has arrived, waiting for confirmation SYN_SENT: The application has started, opening a connection ESTABLISHED: Normal data transmission state FIN_WAIT1: The application says it is done FIN_WAIT2: The other side has agreed to release ITMED_WAIT: Waiting for all packets to die CLOSING: Both sides are trying to close at the same time TIME_WAIT: The other side has initiated a release LAST_ACK: Waiting for all packets to die

The dangers of too much TIME_WAIT

When the network condition is not good, if the active party does not wait for TIME_WAIT, after closing the previous connection, the active party and the passive party establish a new TCP connection. At this time, the FIN packet retransmitted or delayed by the passive party will directly affect the new TCP connection.
Similarly, if the network condition is not good and there is no TIME_WAIT wait, there will be no new connection after the connection is closed. When a retransmitted or delayed FIN packet is received from the passive party, a RST packet will be returned to the passive party, which may affect other service connections of the passive party.

The answer to how to solve the problem of too many TIME_WAIT is as follows:

Edit the kernel file /etc/sysctl.conf and add the following content:

net.ipv4.tcp_syncookies = 1 #Indicates enabling SYN Cookies. When the SYN wait queue overflows, enable cookies to handle it, which can prevent a small amount of SYN attacks. The default value is 0, which means closed;
net.ipv4.tcp_tw_reuse = 1 #Indicates enabling reuse. Allow TIME-WAIT sockets to be reused for new TCP connections. The default value is 0, which means closed.
net.ipv4.tcp_tw_recycle = 1 #Indicates turning on fast recycling of TIME-WAIT sockets in TCP connections. The default value is 0, which means closed.
net.ipv4.tcp_fin_timeout =30#Modify the default TIMEOUT time

Then execute /sbin/sysctl -p to make the parameters take effect.

Simply put, it turns on the system's TIMEWAIT reuse and fast recycling.

More performance optimizations for Linux

If your system already has a large number of connections and the performance is still not ideal after the above configuration tuning, you can optimize the available TCP port range to further improve the server's concurrency capability. Still in the /etc/sysctl.conf file, add the following configuration:

vi /etc/sysctl.conf
#Indicates the frequency at which TCP sends keepalive messages when keepalive is enabled. The default is 2 hours, change it to 20 minutes.
net.ipv4.tcp_keepalive_time = 1200 
# indicates the port range used for outbound connections. The default is very small: 32768 to 61000, changed to 1024 to 65000.
net.ipv4.ip_local_port_range = 1024 65000 
#Indicates the length of the SYN queue. The default is 1024. Increasing the queue length to 8192 can accommodate more network connections waiting to be connected.
net.ipv4.tcp_max_syn_backlog = 8192 
#Indicates the maximum number of TIME_WAIT sockets that the system maintains at the same time. If this number is exceeded, the TIME_WAIT sockets will be cleared immediately and a warning message will be printed. The default value is 180000, change it to 5000. For servers like Apache and Nginx, the parameters in the previous few lines can effectively reduce the number of TIME_WAIT sockets, but for Squid, the effect is not significant. This parameter can control the maximum number of TIME_WAIT sockets to prevent the Squid server from being blocked by a large number of TIME_WAIT sockets.
net.ipv4.tcp_max_tw_buckets = 5000 

More Linux kernel parameter optimization instructions

vim /etc/sysctl.conf

1. net.ipv4.tcp_max_syn_backlog = 65536

The maximum number of recorded connection requests that have not yet received an acknowledgment from the client. For systems with more than 128M of memory, the default value is 1024, and for systems with less than 128M of memory, the default value is 128.

SYN Flood attack exploits the handshake flaws of the TCP protocol, forging false source IP addresses to send a large number of TCP-SYN half-open connections to the target system, eventually causing the target system's socket queue resources to be exhausted and unable to accept new connections. In order to cope with this kind of attack, modern Unix systems generally use multiple connection queue processing to buffer (rather than solve) this kind of attack. One basic queue is used to handle normal fully connected applications (Connect() and Accept()), and another queue is used to store half-open connections separately.

This dual-queue processing method, when used in conjunction with some other system kernel measures (such as Syn-Cookies/Caches), can effectively mitigate small-scale SYN Flood attacks (proven to be <1000p/s). Increasing the SYN queue length can accommodate more network connections waiting to be connected. Generally, websites that are attacked by SYN Flood have a large number of SYN_RECV states, so increasing the tcp_max_syn_backlog value can increase the ability to resist SYN attacks.

2. net.core.netdev_max_backlog = 32768

The maximum number of packets that are allowed to be sent to the queue when each network interface receives packets faster than the kernel can process them.

3. net.core.somaxconn = 32768

Adjust the number of concurrent TCP connections initiated by the system. You may need to increase the connection reserve value to cope with a large number of sudden incoming connection requests. Using a larger value increases the number of pending connections supported, which can reduce the number of connection failures if a large number of connection requests are received simultaneously. A large listen queue can also help prevent DDoS attacks. The maximum number of pending requests is 128 by default.

View the real-time kernel packet loss command:

netstat -su

Location: /proc/sys/

4. net.core.wmem_default = 8388608

This parameter specifies the default value (in bytes) for the send socket buffer size.

5. net.core.rmem_default = 8388608

This parameter specifies the default value (in bytes) for the receive socket buffer size.

6. net.core.rmem_max = 16777216

This parameter specifies the maximum size of the receive socket buffer (in bytes).

7. net.core.wmem_max = 16777216

This parameter specifies the maximum send socket buffer size (in bytes).

8. net.ipv4.tcp_timestamps = 0

Timestamps can prevent forged sequence numbers. A 1G broadband line may re-encounter the old sequence number with an out-of-line value (if it is from the last time it was generated). The timestamp allows the kernel to accept such "abnormal" packets. This needs to be turned off to improve performance.

9. net.ipv4.tcp_synack_retries = 2

For the remote connection request SYN, the kernel will send a SYN+ACK datagram to confirm the receipt of the previous SYN connection request packet. This is the second step of the so-called three-way handshake mechanism. This determines the number of SYN+ACKs the kernel sends before giving up on the connection. Should not be greater than 255. The default value is 5, which corresponds to about 180 seconds. (This value can be determined based on tcp_syn_retries)

10. net.ipv4.tcp_syn_retries = 2

For a new connection, the kernel must send a certain number of SYN connection requests before giving up. Should not be greater than 255. The default value is 5, which corresponds to about 180 seconds. (For networks with heavy loads and good physical communication, this value is too high and can be changed to 2. This value is only for outbound connections. For incoming connections, it is determined by tcp_retries1)

#net.ipv4.tcp_tw_len = 1

11. net.ipv4.tcp_tw_reuse = 1

Indicates enabling reuse, allowing TIME-WAIT Sockets to be reused for new TCP connections. The default value is 0, indicating shutdown. This is very helpful for quickly restarting certain services, but after restarting, it prompts that the port is already in use.

12. net.ipv4.tcp_mem = 94500000 915000000 927000000

tcp_mem has 3 INTEGER variables: low, pressure, high

low: When TCP uses less than this value of memory pages, TCP has no memory pressure and will not consider releasing memory. (Ideally, this value should match the second value specified for tcp_wmem. This second value indicates the maximum page size multiplied by the maximum number of concurrent requests divided by the page size (131072*300/4096)

pressure: When TCP uses more than this value of memory pages, TCP attempts to stabilize its memory usage and enters pressure mode. When the memory consumption falls below the low value, it exits the pressure state. (Ideally this value should be the maximum value of the total buffer size that TCP can use (204800*300/4096)

High: The amount of pages allowed for all TCP Sockets to queue and buffer datagrams. If this value is exceeded, the TCP connection will be rejected, which is why it is not advisable to make it too conservative (512000*300/4096). In this case, the value provided is huge, it can handle 2.5 times more connections than expected; or enable existing connections to carry 2.5 times more data.

Normally these values ​​are calculated at system startup based on the amount of system memory.

13. net.ipv4.tcp_max_orphans = 3276800

The maximum number of TCP sockets that the system can handle that are not owned by any process. If this number is exceeded, connections that do not belong to any process will be reset immediately and a warning message will be displayed. The reason for setting this limit is purely to protect against simple DoS attacks. Do not rely on this or artificially lower this limit.

14. net.ipv4.tcp_fin_timeout = 30

If the socket is closed by the local end, this parameter determines how long it remains in the FIN-WAIT-2 state. The peer can make mistakes and never close the connection, or even crash unexpectedly. The default value is 60 seconds. The usual value for 2.2 kernels is 180 seconds. You can set it this way, but remember that even if your machine is a lightly loaded web server, there is a risk of memory overflow due to a large number of dead sockets. FIN-WAIT-2 is less dangerous than FIN-WAIT-1 because it can only eat up to 1.5K of memory, but their lifetime is longer.

15. net.ipv4.ip_conntrack_max = 10000

Set the system's maximum limit on the number of TCP connections tracked (CentOS 5.6 does not have this parameter)

It also involves a TCP congestion algorithm problem. You can use the following command to view the congestion algorithm control module provided by this machine:

sysctlnet.ipv4.tcp_available_congestion_control

For the analysis of several algorithms, please refer to the following for details: Advantages and disadvantages, applicable environment, and performance analysis of TCP congestion control algorithms. For example, hybla can be tried for high latency, and htcp algorithm can be tried for medium latency.

If you want to set the TCP congestion algorithm to hybla

#Set TCP congestion algorithm net.ipv4.tcp_congestion_control=hybla

For kernel versions higher than 3.7.1, we can enable tcp_fastopen:

# Enable tcp_fastopen
net.ipv4.tcp_fastopen = 3

Iptables related

If not necessary, turn off or uninstall the iptables firewall and prevent the kernel from loading the iptables module. These modules will affect concurrency performance.

IO event allocation mechanism

To enable high concurrent TCP connections in Linux, you must confirm that the application uses the appropriate network I/O technology and I/O event dispatching mechanism. The available I/O technologies are synchronous I/O, non-blocking synchronous I/O, and asynchronous I/O. In the case of high TCP concurrency, if synchronous I/O is used, this will seriously block the operation of the program unless a thread is created for the I/O of each TCP connection. However, too many threads will cause huge overhead due to the system's scheduling of threads. Therefore, it is not advisable to use synchronous I/O in the case of high TCP concurrency. At this time, you can consider using non-blocking synchronous I/O or asynchronous I/O. Non-blocking synchronous I/O techniques include using select(), poll(), epoll and other mechanisms. The technology of asynchronous I/O is to use AIO.

From the perspective of I/O event dispatching mechanism, it is inappropriate to use select() because the number of concurrent connections it supports is limited (usually within 1024). If performance is considered, poll() is not suitable either. Although it can support a higher number of TCP concurrency, it uses a "polling" mechanism. When the number of concurrency is high, its operating efficiency is quite low, and there may be uneven distribution of I/O events, resulting in "starvation" of I/O on some TCP connections. If epoll or AIO is used, there will be no such problem (the AIO technology of the early Linux kernel was implemented by creating a thread in the kernel for each I/O request. This implementation mechanism actually has serious performance problems when used in the case of high-concurrency TCP connections. But in the latest Linux kernel, the implementation of AIO has been improved).

summary

In summary, when developing Linux applications that support high-concurrency TCP connections, you should try to use epoll or AIO technology to implement I/O control on concurrent TCP connections. This will provide effective I/O guarantees for improving the program's support for high-concurrency TCP connections.

After the optimization configuration described above, the server's TCP concurrent processing capability will be significantly improved. The configuration described above is for reference only. If used in a production environment, please adjust and observe according to the actual situation of your own development system deployment.

This is the end of this article about the pitfalls of Linux high concurrency and performance optimization. For more relevant Linux high concurrency and performance optimization content, please search for previous articles on 123WORDPRESS.COM or continue to browse the related articles below. I hope everyone will support 123WORDPRESS.COM in the future!