How to test the maximum number of TCP connections in Linux

Preface

There is a misunderstanding about the maximum number of concurrent connections of a TCP server: "Because the upper limit of the port number is 65535, the maximum number of concurrent connections that a TCP server can theoretically carry is also 65535."

Let me first state the conclusion: for a TCP server process, the number of clients it can connect to simultaneously is not limited by the available port numbers. The number of concurrent connections is limited by the number of files that can be opened in Linux. This number is configurable and can be very large, so it is actually limited by system performance.

Nowadays, if you do server development without high concurrency, you will be embarrassed to go out. So in order to be able to respond righteously when others ask you in the future, "You are improving concurrency every day, what is the highest concurrency you have achieved?", I decided to write a demo myself while I was at home doing nothing during the New Year's Day.

This test is mainly to find out which parameter configurations under Linux limit the maximum number of connections and what the upper limit is.

1. Let me first talk about the idea of ​​demo:

The server is implemented with epoll, which simply receives connections, and the client uses go's goroutine. Each goroutine simply establishes a connection and then does nothing.

Above code:

server:

/*
 * g++ -o test_epoll ./test_epoll.c
 */
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/epoll.h>
#include <netinet/in.h>
#include <arpa/inet.h>

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>

int SetReuseAddr(int fd)
{
 int optval = 1;
 socklen_t optlen = sizeof(optval);
 return setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &optval, optlen);
}

int main()
{
 int fd = socket(AF_INET, SOCK_STREAM, 0);
 int iRet = SetReuseAddr(fd);
 if (iRet != 0)
 {
 printf("setsockopt for SO_REUSEADDR failed, error:%s\n", strerror(iRet));
 return iRet;
 }

 struct sockaddr_in addr;
 memset(&addr, 0, sizeof(addr));
 addr.sin_family = AF_INET;
 addr.sin_port = htons(8080);
 addr.sin_addr.s_addr = INADDR_ANY;
 if (bind(fd, (struct sockaddr*)&addr, sizeof(addr)) == -1)
 {
 printf("bind failed, error:%s\n", strerror(errno));
 return errno;
 }

 if (listen(fd, 5) == -1)
 {
 printf("listen failed, error:%s\n", strerror(errno));
 return errno;
 }
 printf("Listening on 8080...\n");

 int epfd = epoll_create(102400);
 struct epoll_event event;
 event.events = EPOLLIN;
 event.data.fd = fd;
 epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &event);

 struct epoll_event revents[102400];
 int iOnline = 0;
 while (1)
 {
 int num = epoll_wait(epfd, revents, 102400, 60 * 1000);
 printf("epoll_wait return %d\n", num);
 if (num > 0)
 {
  for (int i = 0; i < num; i++)
  {
  if (revents[i].data.fd == fd)
  {
   int client;
   struct sockaddr_in cli_addr;
   socklen_t cli_addr_len = sizeof(cli_addr);
   client = accept(fd, (struct sockaddr*)&cli_addr, &cli_addr_len);
   if (client == -1)
   {
   printf("accept failed, error:%s\n", strerror(errno));
   if (errno == EMFILE)
   {
    printf("per-process limit reached\n");
    exit(errno);
   }
   if (errno == ENFILE)
   {
    printf("system-wide limit reached\n");
    exit(errno);
   }
   continue;
   }

   iOnline++;
   printf("Receive a new connection from %s:%d\n", inet_ntoa(cli_addr.sin_addr), cli_addr.sin_port);
   event.events = EPOLLIN;
   event.data.fd = client;
   epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &event);
  }
  }
 }
 printf("Online number:%d\n", iOnline);
 }

 return 0;
}

client：

package main

import (
 "net"
 "fmt"
 "time"
 "strconv"
 "runtime"
)

func Connect(host string, port int) {
 _, err := net.Dial("tcp", host+":"+strconv.Itoa(port))
 if err != nil {
 fmt.Printf("Dial to %s:%d failed\n", host, port)
 return
 }

 for {
 time.Sleep(30 * 1000 * time.Millisecond)
 }
}

func main() {
 count := 0
 for {
 go Connect("192.168.63.128", 8080)
 count++;
 fmt.Printf("Gorutue num:%d\n", runtime.NumGoroutine())
 time.Sleep(100 * time.Millisecond)
 }
}

2. Start testing

first:

Let me talk about the result first. When the number of connections reached 1031, accept failed. At that time, errno was not judged, so only accept failure was printed out.

Then the first thing that comes to mind is the ulimit -n limit. I checked it and found that the default value is 1024. Then I changed this value and added the following content in /etc/security/limits.conf:

1 * soft nofile 102400
2 * hard nofile 102400

Then close the current xshell connection and reconnect to take effect. Now ulimit -n is 102400.

These two lines mean adjusting the soft and hard limits on the number of file descriptors that each process can open to 102400.

Note: ulimit -n 102400 can also take effect, but this modification is temporary.

Then do a second test.

Second time:

Funny, actually the number of connections is only 2000+. I was wondering why the default number of connections in Windows is so high. It turns out that some connections have been disconnected, but because I didn’t do anything, I thought they were still there. It seems that I have to install another virtual machine [Erha]

To be continued. . .

Install the virtual machine.

Time: 2017-12-31 00:09:00

The virtual machine is installed, then proceed.

This time it really exceeded 10K.

The number of connections is still increasing. I wonder if it can eventually reach 100,000. I am looking forward to it.

Time: 2017-12-31 00:41:00, the final upper limit is stuck at 28232, golang keeps reporting dial failure, because I forgot to print out the specific error message, so I have no idea why dial failed, so I can only run it again T_T

Time: 2017-12-31 01:01:00, added the error message of dial failure, ran it again, and dial failure still occurred at 28232, with the error message:

There is no explanation of the error message in the standard library documentation of golang. From the error message, it seems that the address allocation failed, so I wonder if the port address range is limited.

I checked the port address range and confirmed that this is the limitation. Since the port address is 16 bits, even if the port address range is changed to 1024--65535, a maximum of 64521 connections can be opened. I only have one virtual machine as a client, so it is impossible to achieve 100,000 connections. However, through this test, I also figured out which parameters will limit the upper limit of connections, which is what I want.

Finally, I would like to thank the great people in the Linux kernel team for launching such an awesome mechanism as epoll, which makes it so easy for us to achieve high concurrency. I hope I can be as awesome as them one day, haha.

Summarize

The above is the full content of this article. I hope that the content of this article will have certain reference learning value for your study or work. If you have any questions, you can leave a message to communicate. Thank you for your support for 123WORDPRESS.COM.