TCP socket SYN queue and Accept queue difference analysis

First we must understand that a TCP socket in the "LISTENING" state has two independent queues:

• SYN Queue
• Accept Queue

These two terms are sometimes also called "reqsk_queue", "ACK backlog", "listen backlog", or even "TCP backlog", but we will use the above two terms in this article to avoid confusion.

SYN Queue

The SYN queue stores the connections that receive SYN packets (corresponding to the kernel code structure: struct inet_request_sock). Its responsibility is to reply to SYN+ACK packets and retransmit if no ACK packets are received until it times out. Under Linux, the number of retransmissions is:

$ sysctl net.ipv4.tcp_synack_retries

net.ipv4.tcp_synack_retries = 5

The description of tcp_synack_retries in the documentation is as follows:

tcp_synack_retries - int
The number of times to retransmit SYNACKs for a passive TCP connection. The value cannot exceed 255.
The default value is 5. If the initial RTO is 1 second, the corresponding last retransmission is 31 seconds.
The corresponding last timeout is 63 seconds later.

After sending SYN+ACK, the SYN queue waits for the ACK packet sent from the client (that is, the last packet of the three-way handshake). When an ACK packet is received, the corresponding SYN queue is first found, and then the relevant data in the corresponding SYN queue is checked to see if there is a match. If there is a match, the kernel removes the data related to the connection from the SYN queue, creates a complete connection (corresponding to the kernel code structure: struct inet_sock), and adds this connection to the Accept queue.

Accept Queue

The Accept queue stores established connections, that is, connections waiting to be taken away by the upper-level application. When the process calls accept(), the socket is taken out of the queue and passed to the upper-level application.

This is a brief description of how Linux handles SYN packets. By the way, when the socket has TCP_DEFER_ACCEPT and TCP_FASTOPEN enabled, the working method will be slightly different, which is not introduced in this article.

Queue size limit

The application sets the maximum size of the SYN queue and the Accept queue by calling the listen(2) system call and passing the backlog parameter. For example, as shown below, the maximum sizes of both the SYN queue and the Accept queue are set to 1024:

listen(sfd, 1024)

Note that in kernels prior to 4.3, the size of the SYN queue is calculated in another way.

The maximum size of the SYN queue was previously configured using net.ipv4.tcp_max_syn_backlog, but this is no longer used. Now net.core.somaxconn is used to represent the maximum size of both the SYN queue and the Accept queue. On our servers, we set it to 16k:

$ sysctl net.core.somaxconn

net.core.somaxconn = 16384

After knowing the above information, you may ask, what is the appropriate queue size? What is the appropriate queue size?

The answer is: it depends. For most TCP services this is not too important. For example, before Go language version 1.11, there was no method to set the queue size.

However, there are some legitimate reasons to increase the queue size:

• When the rate of connection establishment requests is really high, the SYN queue may need to be set larger even for a high-performance service.
• The size of the SYN queue, in other words, the number of connections waiting for ACK packets. That is, the longer the average round-trip time with the client, the more connections will accumulate in the SYN queue. For scenarios where most clients are far away from the server, for example, the round-trip time is more than a few hundred milliseconds, the queue size can be set larger.
• If the TCP_DEFER_ACCEPT option is turned on, it will cause the socket to remain in the SYN-RECV state for a longer time, which increases the time it stays in the SYN queue.
  

However, setting the backlog too large can also have adverse effects: each slot in the SYN queue requires some memory. When encountering a SYN Flood attack, there is no need to waste resources on these attacking packets. The inet_request_sock structure in the SYN queue will occupy 256 bytes of memory each under the 4.14 kernel.

Under Linux, if we want to view the current status of the SYN queue, we can use the ss command to query the socket in the SYN-RECV state. For example, the following execution results indicate that there are currently 119 elements in the SYN queue of port 80 and 78 in the SYN queue of port 443.

$ ss -n state syn-recv sport = :80 | wc -l
119
$ ss -n state syn-recv sport = :443 | wc -l
78

What if the program doesn't call accept() fast enough? You can also observe this data through our SystemTap script: resq.stp

What happens if the program doesn't call accept() fast enough?

• Subsequent SYN packets received will not be processed by the SYN queue
• Subsequent ACK packets (used to establish a connection) will not be processed by the SYN queue
• TcpExtListenOverflows / LINUX_MIB_LISTENOVERFLOWS count increases
• TcpExtListenDrops / LINUX_MIB_LISTENDROPS count increases
  

When this happens, we can only hope that the program's processing performance will return to normal later and the client will resend the packets that were discarded by the server.

This behavior of the kernel is acceptable for most services. By the way, you can modify this behavior by adjusting the global parameter net.ipv4.tcp_abort_on_overflow, but it is best not to change this parameter.

You can observe the status of the Accept queue overflow by viewing the count of nstat:

$ nstat -az TcpExtListenDrops
TcpExtListenDrops 49199 0.0

But this is a global count. It is not intuitive to observe. For example, sometimes we observe that it is growing, but all service programs seem to be normal. At this point we can use the ss command to observe the Accept queue size of a single listening port:

$ ss -plnt sport = :6443|cat
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 1024 *:6443 *:*

The Recv-Q column shows the number of sockets in the Accept queue, and Send-Q shows the maximum size of the queue. In the above example, we find that there is no socket that has not been accepted() by the program, but we still find that the ListenDrops count is increasing.

This is because our program is only stuck for a short period of time and does not process new connections, rather than permanently suspending processing. After a while, the program returns to normal. In this case, it is difficult to observe this phenomenon using the ss command, so we wrote a SystemTap script that hooks into the kernel and prints out the discarded SYN packets:

$ sudo stap -v acceptq.stp
time (us) acceptq qmax local addr remote_addr
1495634198449075 1025 1024 0.0.0.0:6443 10.0.1.92:28585
1495634198449253 1025 1024 0.0.0.0:6443 10.0.1.92:50500
1495634198450062 1025 1024 0.0.0.0:6443 10.0.1.92:65434
...

Through the above operations, you can observe which SYN packets are affected by ListenDrops. This way we can also know which programs are losing connections.

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.