Detailed explanation of nginx optimization in high concurrency scenarios

In daily operation and maintenance work, nginx services are often used, and performance bottlenecks caused by high concurrency of nginx are often encountered. Today, I will briefly sort out the configuration of nginx performance optimization (based only on my actual experience, if there is anything wrong, please point it out~)

1. The optimization here mainly refers to the configuration optimization of nginx. Generally speaking, the following items are more effective in the optimization of the nginx configuration file:

1) The number of nginx processes is recommended to be specified according to the number of CPUs, which is usually the same as the number of CPU cores or a multiple of it.

worker_processes 8;

2) Allocate a CPU to each process. In the above example, 8 processes are assigned to 8 CPUs. Of course, you can write more, or assign one process to multiple CPUs.

worker_cpu_affinity 00000001 00000010 00000100 00001000 000100000 001000000 01000000 10000000;

3) The following command refers to the maximum number of file descriptors opened by an nginx process. The theoretical value should be the maximum number of open files in the system (ulimit -n) divided by the number of nginx processes. However, nginx does not distribute requests evenly, so it is best to keep it consistent with the value of ulimit -n.

worker_rlimit_nofile 65535;

4) Use the epoll I/O model to efficiently handle asynchronous events

use epoll;

5) The maximum number of connections allowed for each process. Theoretically, the maximum number of connections for each nginx server is worker_processes*worker_connections.

worker_connections 65535;

6) HTTP connection timeout, the default is 60s, the function is to make the connection from the client to the server remain valid within the set time, when there is a subsequent request to the server, this function avoids establishing or re-establishing the connection. Remember not to set this parameter too large! Otherwise, many invalid http connections will occupy the number of nginx connections, and eventually nginx will crash!

keepalive_timeout 60;

7) The buffer size of the client request header can be set according to your system paging size. Generally, the header size of a request will not exceed 1k. However, since the system paging is generally larger than 1k, it is set to the paging size here. The paging size can be obtained using the command getconf PAGESIZE.

client_header_buffer_size 4k;

8) The following parameter will specify the cache for open files. It is not enabled by default. max specifies the number of caches, which is recommended to be consistent with the number of open files. inactive refers to the time after which the cache is deleted if the file is not requested.

open_file_cache max=102400 inactive=20s;

9) The following refers to how often to check for valid cache information.

open_file_cache_valid 30s;

10) The minimum number of times a file is used within the inactive parameter time in the open_file_cache directive. If this number is exceeded, the file descriptor is always opened in the cache. As in the above example, if a file is not used once within the inactive time, it will be removed.

open_file_cache_min_uses 1;

11) Hide the information about the operating system and web server (Nginx) version number in the response header, which is good for security.

server_tokens off;

12) Allow sendfile() to work. sendfile() can copy data (or any two file descriptors) between disk and a TCP socket. Pre-sendfile is to apply for a data buffer in user space before transmitting data. Then use read() to copy the data from the file to this buffer, and write() to write the buffer data to the network. sendfile() immediately reads data from disk to the OS cache. Because the copying is done in the kernel, sendfile() is more efficient than combining read() and write() and turning discard buffering on and off (more on sendfile in a bit).

sendfile on;

13) Tell nginx to send all headers in one packet instead of sending them one by one. That is to say, the data packet will not be transmitted immediately. It will be transmitted all at once when the data packet is the largest. This will help solve network congestion.

tcp_nopush on;

14) Tell nginx not to cache data, but to send it piece by piece - when you need to send data in a timely manner, you should set this attribute for the application, so that when sending a small piece of data information, you cannot get the return value immediately.

tcp_nodelay on;

for example:

http { 
server_tokens off; 
sendfile on; 
tcp_nopush on; 
tcp_nodelay on; 
......
} 

15) The buffer size of the client request header can be set according to the system paging size. Generally, the size of a request header will not exceed 1k. However, since the system paging is generally larger than 1k, it is set to the paging size here.

client_header_buffer_size 4k;

The buffer size of the client request header can be set according to the system paging size. Generally, the size of a request header will not exceed 1k. However, since the system paging is generally larger than 1k, it is set to the paging size here.
The paging size can be obtained using the command getconf PAGESIZE.

[root@test-huanqiu ~]# getconf PAGESIZE 
4096

However, there are cases where client_header_buffer_size exceeds 4k. However, the value of client_header_buffer_size must be set to an integer multiple of the "system paging size".

16) Specify cache for open files. It is not enabled by default. max specifies the number of caches. It is recommended to be consistent with the number of open files. inactive refers to the time after which the cache is deleted if the file is not requested.

open_file_cache max=65535 inactive=60s;

17) The minimum number of times a file is used within the inactive parameter time in the open_file_cache directive. If this number is exceeded, the file descriptor is always opened in the cache. As in the above example, if a file is not used once within the inactive time, it will be removed.

open_file_cache_min_uses 1;

18) Specify how often to check cached information for validity.

open_file_cache_valid 80s;

The following is a simple nginx configuration file that I use:

[root@dev-huanqiu ~]# cat /usr/local/nginx/conf/nginx.conf
user www www;
worker_processes 8;
worker_cpu_affinity 00000001 00000010 00000100 00001000 000100000 001000000;
error_log /www/log/nginx_error.log crit;
pid /usr/local/nginx/nginx.pid;
worker_rlimit_nofile 65535;
 
events
{
  use epoll;
  worker_connections 65535;
}
 
http
{
  include mime.types;
  default_type application/octet-stream;
 
  charset utf-8;
 
  server_names_hash_bucket_size 128;
  client_header_buffer_size 2k;
  large_client_header_buffers 4 4k;
  client_max_body_size 8m;
 
  sendfile on;
  tcp_nopush on;
 
  keepalive_timeout 60;
 
  fastcgi_cache_path /usr/local/nginx/fastcgi_cache levels=1:2
         keys_zone=TEST:10m
         inactive=5m;
  fastcgi_connect_timeout 300;
  fastcgi_send_timeout 300;
  fastcgi_read_timeout 300;
  fastcgi_buffer_size 16k;
  fastcgi_buffers 16 16k;
  fastcgi_busy_buffers_size 16k;
  fastcgi_temp_file_write_size 16k;
  fastcgi_cache TEST;
  fastcgi_cache_valid 200 302 1h;
  fastcgi_cache_valid 301 1d;
  fastcgi_cache_valid any 1m;
  fastcgi_cache_min_uses 1;
  fastcgi_cache_use_stale error timeout invalid_header http_500; 
  open_file_cache max=204800 inactive=20s;
  open_file_cache_min_uses 1;
  open_file_cache_valid 30s; 
 
  tcp_nodelay on;
  
  gzip on;
  gzip_min_length 1k;
  gzip_buffers 4 16k;
  gzip_http_version 1.0;
  gzip_comp_level 2;
  gzip_types text/plain application/x-javascript text/css application/xml;
  gzip_vary on;
 
  server
  {
   listen 8080;
   server_name huan.wangshibo.com;
   index index.php index.htm;
   root /www/html/;
 
   location /status
   {
     stub_status on;
   }
 
   location ~ .*\.(php|php5)?$
   {
     fastcgi_pass 127.0.0.1:9000;
     fastcgi_index index.php;
     include fcgi.conf;
   }
 
   location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|js|css)$
   {
    expires 30d;
   }
 
   log_format access '$remote_addr - $remote_user [$time_local] "$request" '
        '$status $body_bytes_sent "$http_referer" '
        '"$http_user_agent" $http_x_forwarded_for';
   access_log /www/log/access.log access;
    }
}

2. Several instructions about FastCGI

1) This directive specifies a path, directory structure level, key area storage time and inactive deletion time for the FastCGI cache.

fastcgi_cache_path /usr/local/nginx/fastcgi_cache levels=1:2 keys_zone=TEST:10m inactive=5m;

2) Specify the timeout for connecting to the backend FastCGI.

fastcgi_connect_timeout 300;

3) The timeout for sending a request to FastCGI. This value refers to the timeout for sending a request to FastCGI after two handshakes have been completed.

fastcgi_send_timeout 300;

4) The timeout for receiving FastCGI responses. This value refers to the timeout for receiving FastCGI responses after completing two handshakes.

fastcgi_read_timeout 300;

5) Specify the size of the buffer needed to read the first part of the FastCGI response. Here you can set it to the buffer size specified by the fastcgi_buffers directive. The above directive specifies that it will use a 16k buffer to read the first part of the response, that is, the response header. In fact, this response header is generally very small (no more than 1k), but if you specify the buffer size in the fastcgi_buffers directive, it will also allocate a buffer size specified by fastcgi_buffers to cache.

fastcgi_buffer_size 16k;

6) Specify how many and how large the local buffer is needed to buffer FastCGI's response. As shown above, if the page size generated by a PHP script is 256k, 16 16k buffers will be allocated for caching. If it is larger than 256k, the part larger than 256k will be cached in the path specified by fastcgi_temp. Of course, this is an unwise solution for server load, because the data processing speed in memory is faster than the hard disk. Usually, this value should be set to an intermediate value of the page size generated by the PHP scripts in your site. For example, if the page size generated by most scripts in your site is 256k, you can set this value to 16 16k, or 4 64k or 64 4k, but obviously, the latter two are not good settings, because if the generated page is only 32k, if 4 64k is used, it will allocate 1 64k buffer to cache, and if 64 4k is used, it will allocate 8 4k buffers to cache, and if 16 16k is used, it will allocate 2 16k to cache the page, which seems more reasonable.

fastcgi_buffers 16 16k;

7) I don’t know what this directive is used for, I only know that the default value is twice fastcgi_buffers.

fastcgi_busy_buffers_size 32k;

8) How big the data block will be when writing to fastcgi_temp_path. The default value is twice fastcgi_buffers.

fastcgi_temp_file_write_size 32k;

9) Enable FastCGI cache and assign a name to it. Personally, I feel that enabling cache is very useful, as it can effectively reduce CPU load and prevent 502 errors. But this cache can cause a lot of problems because it caches dynamic pages. The specific use depends on your needs.

fastcgi_cache TEST

10) Specify a cache time for a specific response code. For example, in the above example, 200 and 302 responses are cached for one hour, 301 responses are cached for one day, and the others are cached for one minute.

fastcgi_cache_valid 200 302 1h;
fastcgi_cache_valid 301 1d;
fastcgi_cache_valid any 1m;

11) The minimum number of times the cache is used within the inactive parameter value time of the fastcgi_cache_path directive. For example, if a file is not used once within 5 minutes, the file will be removed.

fastcgi_cache_min_uses 1;

12) I don't know the function of this parameter, but I guess it should let nginx know which types of cache are useless.

fastcgi_cache_use_stale error timeout invalid_header http_500;

The above are the FastCGI related parameters in nginx.

In addition, FastCGI itself also has some configurations that need to be optimized. If you use php-fpm to manage FastCGI, you can modify the following values ​​in the configuration file:

1) The number of concurrent requests processed at the same time, that is, it will open up to 60 child threads to handle concurrent connections.

<value name="max_children">60</value>

2) Maximum number of open files.

<value name="rlimit_files">65535</value>

3) The maximum number of requests each process can execute before being reset.

<value name="max_requests">65535</value>

3. Optimization of kernel parameters in the /etc/sysctl.conf file

1) The number of timewait, the default is 180000. (Deven: Therefore, if you want to reduce timewait, you need to reduce the value of tcp_max_tw_buckets)

net.ipv4.tcp_max_tw_buckets = 6000

2) The port range that the system is allowed to open.

net.ipv4.ip_local_port_range = 1024 65000

3) Enable the fast recycling function of TIME-WAIT state sockets; used to quickly reduce the number of TCP connections in TIME-WAIT state. 1 means enabled; 0 means disabled. However, please note that it is generally not recommended to enable this option because it will cause a large number of TCP connection establishment errors under NAT (Network Address Translation) networks, thus causing website access failures.

net.ipv4.tcp_tw_recycle = 0

In fact, the net.ipv4.tcp_tw_recycle function needs to be enabled after the net.ipv4.tcp_timestamps switch is enabled (usually the system enables this function by default).

When tcp_tw_recycle is turned on (tcp_timestamps is turned on at the same time, the effect of fast socket recycling is achieved), it is a disaster for the client behind the NAT device!

This will cause the client connection to the server behind the NAT device to be unstable (some clients can connect to the server, and some clients cannot connect to the server).
In other words, the tcp_tw_recycle function is designed for internal networks (network environment is self-controllable) - when NAT does not exist, and is not suitable for use in public networks.

Generally speaking, the reason for recycling a socket in TIME_WAIT state is that it is "unable to actively connect to the remote end" because there are no available ports, and it should not be to recycle memory (which is unnecessary).

That is: the demand is the client's demand, will the server have the problem of "insufficient ports"?

Unless it is a front-end machine, it needs a lot of connections to the back-end services, that is, it plays the role of a client.

The correct solution to this problem should always be:

net.ipv4.ip_local_port_range = 9000 6553 #The default value is small net.ipv4.tcp_max_tw_buckets = 10000 #The default value is small and can be adjusted appropriately net.ipv4.tcp_tw_reuse = 1 
net.ipv4.tcp_fin_timeout = 10

4) Enable the reuse function to allow sockets in TIME-WAIT state to be reused for new TCP connections. It is safe to enable this function, so generally do not change it!

net.ipv4.tcp_tw_reuse = 1

5) Enable SYN Cookies. When the SYN wait queue overflows, enable cookies to handle it.

net.ipv4.tcp_syncookies = 1

6) The backlog of the listen function in the web application will limit our kernel parameter net.core.somaxconn to 128 by default, and the default value of NGX_LISTEN_BACKLOG defined by nginx is 511, so it is necessary to adjust this value.

net.core.somaxconn = 262144

7) The maximum number of packets allowed to be sent to the queue when the rate at which each network interface receives packets is faster than the rate at which the kernel can process them.
net.core.netdev_max_backlog = 262144

8) What is the maximum number of TCP sockets in the system that are not associated with any user file handle? If this number is exceeded, orphaned connections will be reset immediately and a warning message will be printed. This limit is only to prevent simple DoS attacks. Do not rely too much on it or artificially reduce this value. Instead, you should increase this value (if you increase the memory).

net.ipv4.tcp_max_orphans = 262144

9) The maximum number of connection requests recorded that have not yet received a client acknowledgment. For systems with 128M of memory, the default value is 1024, and for systems with less memory it is 128.

net.ipv4.tcp_max_syn_backlog = 262144

10) Timestamps can avoid sequence number wraparound. A 1Gbps link will certainly encounter sequence numbers that have been used before. The timestamp allows the kernel to accept such "abnormal" packets.

net.ipv4.tcp_timestamps = 1

In order to improve performance, many servers enable the net.ipv4.tcp_tw_recycle option. In a NAT network environment, this may cause some connection failures in website access.

Personal advice:

Turn off net.ipv4.tcp_tw_recycle option instead of net.ipv4.tcp_timestamps;

Because when net.ipv4.tcp_timestamps is turned off, turning on net.ipv4.tcp_tw_recycle will not work; however, net.ipv4.tcp_timestamps can be turned on independently and work.

11) In order to open the connection to the other end, the kernel needs to send a SYN and an ACK in response to the previous SYN. This is the second handshake in the so-called three-way handshake. This setting determines the number of SYN+ACK packets the kernel sends before giving up on the connection.

net.ipv4.tcp_synack_retries = 1

12) The number of SYN packets sent before the kernel gives up on establishing a connection.

net.ipv4.tcp_syn_retries = 1

13) If the socket is closed by the local end, this parameter determines how long it remains in the FIN-WAIT-2 state. The peer can make mistakes and never close the connection, or even crash unexpectedly. The default value is 60 seconds. The usual value for 2.2 kernels is 180 seconds. You can set it this way, but remember that even if your machine is a lightly loaded web server, there is a risk of memory overflow due to a large number of dead sockets. FIN-WAIT-2 is less dangerous than FIN-WAIT-1 because it can only eat up to 1.5K of memory, but their lifetime is longer.

net.ipv4.tcp_fin_timeout = 30

14) When keepalive is enabled, the frequency at which TCP sends keepalive messages. The default is 2 hours.

net.ipv4.tcp_keepalive_time = 30

Below is a standard configuration of kernel parameters that I often use.

[root@dev-huanqiu ~]# cat /etc/sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1 //These four lines marked in red are usually the solution when a large number of TIME_WAIT are found kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.tcp_max_tw_buckets = 6000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_timestamps = 1 //When net.ipv4.tcp_tw_recycle is set to 1, this option is best combined with net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_tw_recycle = 1 // Enabling this function can reduce the TIME-WAIT state, but it may cause TCP connection errors in NAT network mode, so be careful.
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 30
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.ip_conntrack_max = 6553500

-------------------------------------Record of a small accident----------------------------------------------------

net.ipv4.tcp_tw_recycle = 1 After this function is turned on, it can indeed reduce the TIME-WAIT state. I usually turn on this parameter.

But I also stepped on a pit because of this parameter:

The company's CMS backend system for publishing news uses the haproxy+keepalived proxy architecture, and all the external IP addresses of the real servers on the backend are removed.

Phenomenon: During the peak posting period one morning, there was an access failure in the CMS backend. Restarting the PHP service would immediately take effect, but after a period of time, the access would fail again.

I checked the nginx and php logs but didn't find anything. Later I googled it and found that it was the net.ipv4.tcp_tw_recycle parameter that caused the problem!

This network architecture is in NAT mode for the backend realserver. When this parameter is turned on, a large number of TCP connection establishment errors will occur, causing website access failures.

Finally, set net.ipv4.tcp_tw_recycle to 0. After turning off this function, background access will immediately return to normal.

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.