TCP performance tuning implementation principle and process analysis

Three-way handshake phase

Number of retries for client SYN packets

sysctl -w net.ipv4.tcp_syn_retries=6

Related Introduction

The first retry occurs after 1 second, followed by a total of 6 retries at 2, 4, 8, 16, and 32 seconds in a doubling manner. The last retry will wait for 64 seconds. If there is still no ACK returned, the three-way handshake will be terminated. Therefore, the total time is 1+2+4+8+16+32+64=127 seconds, which is more than 2 minutes.

Server semi-connection pool size

sysctl -w net.ipv4.tcp_max_syn_backlog=16384

Whether to enable the syncookie mechanism after the server semi-connection pool is full

sysctl -w net.ipv4.tcp_syncookies=1

Related Introduction

If the SYN half-connection queue is full, the connection will be dropped by default. This is not the case. By turning on the syncookies function, the connection can be successfully established without using the SYN queue.

Syncookies works like this: the server calculates a value based on the current state and puts it in the SYN+ACK message it sends. When the client returns the ACK message, the value is taken out for verification. If it is legal, the connection is considered to be established successfully, as shown in the figure below.

• 0 means turning off this function;
• 2 means unconditionally enable the function;
• 1 means that it will be enabled only when the SYN semi-connection queue is full.

Note: Since syncookie is only used to deal with SYN flood attacks (attackers maliciously construct a large number of SYN messages and send them to the server, causing the SYN semi-connection queue to overflow, resulting in the inability to establish normal client connections), many TCP features cannot be used for connections established in this way. Therefore, you should set tcp_syncookies to 1 and enable it only when the queue is full.

Number of retries for the server's SYN+ACK packet

net.ipv4.tcp_synack_retries=5

Related Introduction

The default retry count of tcp_synack_retries is 5 times, which is similar to the client resending SYN. ​​Its retries will go through 1, 2, 4, 8, and 16 seconds. After the last retry, it will wait for 32 seconds. If ACK is still not received, the connection will be closed, so it takes a total of 63 seconds to wait.

The size of the server's full connection queue

Depends on min(backlog, /proc/sys/net/core/somaxconn). After Linux kernel version 2.2, the backlog parameter of the listen function can set the size of the accept queue.

In addition, the backlog parameter is also limited by the Linux system-level queue length upper limit. Of course, this upper limit threshold can also be modified by the somaxconn parameter. somaxconn is a kernel parameter and the default value is 128.

sysctl -w net.core.somaxconn=32768

Four wave phases

Next, we call the party that closes the connection first the active party, and the party that closes the connection later the passive party.

The process of four waves:

In fact, the four waves only involve two types of messages: FIN and ACK. FIN means Finish, which means ending the connection. Whoever sends a FIN message means that it will no longer send any data and close the transmission channel in this direction. ACK stands for Acknowledge, which is used to notify the other party that your sending channel has been closed. When the active party closes the connection, it sends a FIN message, and the connection state of the active party changes from ESTABLISHED to FIN_WAIT1. When the passive party receives the FIN message, the kernel automatically replies with an ACK message, and the connection state changes from ESTABLISHED to CLOSE_WAIT. As the name suggests, it is waiting for the process to call the close function to close the connection. When the active party receives this ACK message, the connection state changes from FIN_WAIT1 to FIN_WAIT2, and the active party's sending channel is closed. Now let's look at how the passive party's sending channel is closed. When the passive party enters the CLOSE_WAIT state, the read function of the process will return 0, so the developer will call the close function in a targeted manner, which will trigger the kernel to send a FIN message. At this time, the state of the passive party connection becomes LAST_ACK. When the active party receives this FIN message, the kernel will automatically reply ACK, and the connection status will change from FIN_WAIT2 to TIME_WAIT. Under the Linux system, the connection in the TIME_WAIT state will be completely closed after about 1 minute. When the passive party receives the ACK message, the connection will be closed.

Active side optimization

Waiting for ACK, FIN packet retransmission times

After the active party sends a FIN message, the connection is in the FIN_WAIT1 state, which should usually be converted to FIN_WAIT2 within tens of milliseconds. Only when the ACK returned by the other party is not received for a long time, can the FIN_WAIT1 state be observed using the netstat command. At this point, the kernel will periodically resend the FIN message, where the number of retransmissions is controlled by the tcp_orphan_retries parameter (note that although orphan means orphan, this parameter is not only valid for orphan connections, in fact, it is valid for all connections in the FIN_WAIT1 state). The default value is 0, specifically 8 times:

net.ipv4.tcp_orphan_retries = 0

The number of orphan connections

net.ipv4.tcp_max_orphans = 16384

Related Introduction

tcp_max_orphans defines the maximum number of orphan connections. When the process calls the close function to close the connection, the connection is in the FIN_WAIT1 state. This connection has nothing to do with the process and becomes an orphan connection. In order to prevent too many orphan connections from causing system resources to be occupied for a long time, the Linux system provides the tcp_max_orphans parameter. If the number of orphan connections is greater than it, the newly added orphan connections will no longer go through four waves, but will directly send an RST reset message to forcibly close.

Definition of orphan connection: A connection closed by a process calling close is called an orphan connection. In addition, the shutdown function can also close a connection. Both of them will send a FIN message to the other party (the shutdown parameter must be passed in SHUT_WR or SHUT_RDWR to send FIN). The difference is that after the close call, even if the data sent by the other party in the semi-closed state reaches the active party, the process cannot receive it. If you use the netstat -p command, you will find that the process name corresponding to the connection is empty (it has nothing to do with the process!). After the shutdown function is called, even if the connection enters the FIN_WAIT1 or FIN_WAIT2 state, it is not an orphan connection and the process can continue to receive data.

Waiting time for FIN

net.ipv4.tcp_fin_timeout = 60

Related Introduction

When the connection receives ACK and enters the FIN_WAIT2 state, it means that the sending channel of the active party has been closed. Next, it will wait for the other party to send a FIN message and close the other party's sending channel. At this time, if the connection is closed using the shutdown function, the connection can remain in the FIN_WAIT2 state. But for orphan connections closed by the close function, this state cannot last too long, and tcp_fin_timeout controls how long the connection lasts in this state.

TIME_WAIT related parameters

Related Introduction

TIME_WAIT is the last state of the active party's four waves. When receiving the FIN message from the passive party, the active party replies ACK, confirming that the other party's sending channel has been closed. The connection then enters the TIME_WAIT state and is closed after waiting for 60 seconds.

Maximum number of connections in TIME_WAIT state

When the number of TIME_WAIT connections exceeds this parameter, the newly closed connection will no longer go through TIME_WAIT and will be closed directly.

net.ipv4.tcp_max_tw_buckets = 5000

Whether to reuse ports in TIME_WAIT state

To reuse ports in the TIME_WAIT state, if the server actively initiates connections to upstream servers, you can set the tcp_tw_reuse parameter to 1, which allows new connections as clients to use ports in the TIME_WAIT state under safe conditions.

net.ipv4.tcp_tw_reuse = 1

Of course, in order for tcp_tw_reuse to take effect, the timestamps parameter must be set to 1, which satisfies the prerequisite for safe reuse (the other party must also turn on tcp_timestamps):

net.ipv4.tcp_timestamps = 1

Old versions of Linux also provide the tcp_tw_recycle parameter, which does not require the TIME_WAIT state to exist for 60 seconds, which can easily lead to data confusion. It is not recommended to set it to 1.

net.ipv4.tcp_tw_recycle = 0

Therefore, after Linux version 4.12, this parameter was directly cancelled.

Other Configuration

The port range allowed to be opened by the system

sysctl -w net.ipv4.ip_local_port_range=1024 65000

The maximum number of file handles that the system globally allows to allocate

sysctl -w fs.file-max=2097152

sysctl -w fs.nr_open=2097152

echo 2097152 > /proc/sys/fs/nr_open

The number of open file handles allowed for the current session or process

ulimit -n 1048576

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.