Troubleshooting the cause of 502 bad gateway error on nginx server

The server reports an error 502 when synchronizing the public account fan data and batch pushing

According to the error message, it can be determined that it is a backend problem. There are many reasons for 502 errors, but generally speaking, the server cannot handle it.

1. First check the server log
1) Check the nginx log first. If you are not familiar with it, you can get the path of error_log from nginx.conf and find the error as follows:

It is found that the number of connections handled by the nginx process is not enough, and the number of connections handled by a single process exceeds the worker_connections value configured in nginx.conf

Usually the value of worker_connections can refer to the maximum number of connections opened by a single process. The command is: ulimit -n
ulimit -a View all limit parameters. The current maximum number of open files is 65535. You can set worker_connections to 51200.

Restart nginx nginx -s reload

View the current TCP connection status netstat -an|awk '/^tcp/{++S[$NF]}END{for (a in S)print a,S[a]}'

2) Check the php-fpm log. If you are not familiar with the log location, you can check it from php-fpm.conf. Note that the configuration file for php7 and above is placed in the www.conf directory under the php-fpm.d directory.

View php-fpm.log

It is found that pm.max_children is not enough, indicating that the maximum number of php-fpm processes is too small. Check the php configuration file ww.conf and modify the parameter pm.max_children=100

php-fpm mainly adjusts several parameters
pm = dynamic How to control the child process, options are static and dynamic

pm.max_children: The number of php-fpm processes opened in static mode pm.max_requests: The maximum number of requests that a php-fpm child process can handle pm.start_servers: The number of starting php-fpm processes in dynamic mode pm.min_spare_servers: The minimum number of php-fpm processes in dynamic mode pm.max_spare_servers: The maximum number of php-fpm processes in dynamic mode

1. What is the appropriate value for pm.max_children and pm.max_spare_servers?

In principle, the larger this value is, the better. If there are more php-cgi processes, the processing will be faster and there will be fewer queued requests.
Setting "max_children" also needs to be set according to the performance of the server.

The number can also be obtained based on memory/30M. For example, if the memory is 8GB, it can be set to 100, so the memory consumed by php-fpm can be controlled at around 2G-3G.

For a server with small memory, such as a VPS with 256M memory, even if you calculate it based on a 20M memory capacity, 10 php-cgi processes will consume 200M memory, so it is normal for the system to crash.

Therefore, we should try to control the number of php-fpm processes and roughly understand the memory occupied by other applications.
Assigning it a small static amount will make the system more stable. Or use dynamic method,
Because the dynamic method will end unnecessary processes and reclaim some memory, it is recommended to use it on servers or VPSs with less memory. The specific maximum number is obtained based on memory/30M.

The default value calculation formula of pm.start_servers is: min_spare_servers + (max_spare_servers - min_spare_servers) / 2.

For example, if you have a 512M VPS and allocate a maximum of 250M to php-fpm, it is recommended that you set pm.max_spare_servers to 250/30, which is about 8. As for pm.min_spare_servers, it is recommended to set it according to the server load. For example, if only the PHP environment is deployed on the server, the more appropriate value is between 2 and 5.

There is another problem here. php-fpm may cause memory leaks due to some third-party libraries. Over time, it will occupy more memory. For example, our server now occupies about 50m. Fortunately, there is the pm.max_requests parameter, which specifies how many times a php-fpm child process is executed before restarting the process. This may need to be adjusted according to your actual situation.

The calculation is as follows:

Generally speaking, under normal circumstances, the memory consumed by each PHP-CGI on a server is around 20M~30M, so I set my "max_children" to 40, 20M*40=800M, which means that at peak times, the memory consumed by all PHP-CGIs is within 800M, which is lower than my effective memory of 2Gb.

If my "max_children" is set to a smaller value, such as 5-10, then php-cgi will be "very tired", the processing speed will be very slow, the waiting time will be long, and the CPU usage will be high.

If a request is not processed for a long time, the 504 Gateway Time-out error will appear, and if the php-cgi that is being processed encounters a problem, the 502 Bad gateway error will appear.

The best way to set max_children is based on req/s
(throughput, the maximum number of requests processed by the server per unit time, unit req/s) to set,
If the program has a processing capacity of 100 req/s, then it is better to set it to 100. This is adjusted dynamically.

2. What is the appropriate request_terminate_timeout value?

The calculation is as follows:

If your server performance is good enough, the bandwidth resources are sufficient, and the PHP script has no loops or bugs, you can directly set "request_terminate_timeout" to 0s. 0s means that PHP-CGI will continue to execute without time limit.

If you can't do this, that is to say, your PHP-CGI may have a bug, or your bandwidth is not sufficient or other reasons cause your PHP-CGI to hang, then it is recommended that you assign a value to "request_terminate_timeout", which can be set according to the performance of your server.

Generally speaking, the better the performance, the higher you can set it, 20 minutes to 30 minutes is fine. Since my server PHP scripts need to run for a long time, some may exceed 10 minutes, so I set 900 seconds, which will not cause PHP-CGI to die and cause the 502 Bad gateway error.

Optimized parameters

Edit /usr/local/php/etc/php-fpm.d/www.conf:
Server configuration: 2 cores 8G
pm = dynamic
pm.start_servers = 5
pm.min_spare_servers = 2
pm.max_spare_servers = 100

request_terminate_timeout=1200

Find out the process id of the service
ps aux |grep php-fpm
kill -9 process id is often used to kill zombie processes

Summarize the causes of 502 errors in nginx

2. The proxy buffer is too small. If you are using nginx reverse proxy, if the header is too large, exceeding the default 1k, it will cause the upstream sent too big header mentioned above (to put it bluntly, nginx sends the external request to the backend for processing, and the header returned by the backend is too large, which nginx cannot process, resulting in 502.

server {

listen 80;
server_name *.lxy.me;

location / {

Add these 3 lines
fastcgi_buffer_size 64k;
fastcgi_buffers 32 32k;
fastcgi_busy_buffers_size 128k;
Add these 3 lines
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
…………
}

3. The default number of php-cgi processes is set too small. 502 problems may occur during installation and use. Generally, this is because the default number of php-cgi processes is 5. 502 may be caused by insufficient phpcgi processes. You need to modify /usr/local/php/etc/php-fpm.conf and increase the max_children value appropriately. It is also possible that the max_requests value is not enough. It should be noted that these two configuration items take up a lot of memory, please set them according to the server configuration. Otherwise it may have the opposite effect.

4. PHP execution timeout
PHP execution timeout, modify /usr/local/php/etc/php.ini and change max_execution_time to 300

5. Nginx waiting time timeout The execution time of some PHP programs exceeds the Nginx waiting time. You can appropriately increase the FastCGI timeout time in the nginx.conf configuration file

http {
fastcgi_connect_timeout 300;
fastcgi_send_timeout 300;
fastcgi_read_timeout 300;
…
}

6. If you are working on a public account, please note that it may be caused by too many requests from the WeChat server to your own server. If you find that the number of php-fpm processes has reached the maximum number of processes, check the php-fpm configuration file and you will see an error message.

This is the end of this article about troubleshooting the cause of nginx server exception 502 bad gateway. For more related nginx server exception 502 bad gateway content, please search 123WORDPRESS.COM's previous articles or continue to browse the following related articles. I hope everyone will support 123WORDPRESS.COM in the future!