Detailed explanation of Nginx status monitoring and log analysis

1. Nginx status monitoring

Nginx provides a built-in status information monitoring page that can be used to monitor the overall access status of Nginx. This function is implemented by the ngx_http_stub_status_module module.

Use the nginx -V 2>&1 | grep -o with-http_stub_status_module command to check whether the current Nginx has the status function. If the output is ngx_http_stub_status_module, it means that it has the status function. If not, you can add this module during compilation.

By default, status is disabled, we need to enable it and specify the uri to access the data.

server {
  listen 80;
  server_name default_server;
  location /status {
    stub_status on; 
    allow 114.247.125.227;
  }
}

The allow configuration only allows the specified IP to access the nginx status function. If it is removed, there is no restriction.

After restarting Nginx, visit http://{IP}/status in your browser to view status monitoring information.

• Active connections: The current number of active client connections (including waiting client connections), equivalent to the TCP connection status in Established and SYN_ACK
• accepts: The total number of accepted client connections, that is, the connections that have been received by the worker process
• handled: The total number of connections that have been handled
• requests: total number of http requests from the client
• Reading: The number of http requests currently being read (read the http request header)
• Writing: The number of connections currently ready to respond (written to the http response header)
• Waiting: The number of idle client requests currently waiting. The waiting time is the interval between Reading and Writing.
  

After collecting Nginx data, you can use monitoring tools to monitor it.

2. Log analysis

Nginx default log format configuration can be found in /etc/nginx/nginx.conf

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
           '$status $body_bytes_sent "$http_referer" '
           '"$http_user_agent" "$http_x_forwarded_for" $request_time $upstream_response_time';

Printed log example

39.105.66.117 - mp [11/Sep/2019:19:03:01 +0800] "POST /salesplatform-gateway/users HTTP/1.1" 200 575 "-" "Apache-HttpClient/4.5.5 (Java/1.8.0_161)" "-" 0.040 0.040
39.105.66.117 - mp [11/Sep/2019:19:03:08 +0800] "POST /salesplatform-gateway/users HTTP/1.1" 200 575 "-" "Apache-HttpClient/4.5.5 (Java/1.8.0_161)" "-" 0.008 0.008

• $remote_addr: client's IP address
• $remote_user: Used to record the user name of the remote client
• $time_local: used to record access time and time zone
• $request: used to record the request URL and request method
• $status: response status code
• $body_bytes_sent: The number of bytes of the file body content sent to the client
• $http_referer: can record the link from which the user accessed
• $http_user_agent: information about the browser used by the user
• $http_x_forwarded_for: can record the client IP and record the client's IP address through the proxy server
• $request_time: refers to the time from receiving the first byte of the user's request to sending the response data. That is, $request_time includes the time of receiving the client's request data, the time of the backend program responding, and the time of sending the response data to the client.
• $upstream_response_time: The time used to receive the response from the upstream server
  

Common analysis commands

1. Count UV based on access IP

awk '{print $1}' paycenteraccess.log | sort -n | uniq | wc -l

2. Query the most frequently visited IPs (top 10)

awk '{print $1}' /var/log/nginx/access.log | sort -n | uniq -c | sort -rn | head -n 10

3. Check the IP access volume in a certain period of time (1-8 o'clock)

awk '$4 >="[25/Mar/2020:01:00:00" && $4 <="[25/Mar/2020:08:00:00"' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -nr |wc -l

4. View IP addresses that have visited more than 100 times

awk '{print $1}' /var/log/nginx/access.log | sort -n |uniq -c |awk '{if($1 >100) print $0}'|sort -rn

5. View the URLs and number of visits visited by the specified IP

grep "39.105.67.140" /var/log/nginx/access.log|awk '{print $7}' |sort |uniq -c |sort -n -k 1 -r

6. Count PV based on visited URLs

cat /var/log/nginx/access.log |awk '{print $7}' |wc -l

7. Query the most frequently visited URLs (top 10)

awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -n 10

8. View the most frequently visited URLs ([excluding /api/appid]) (top 10)

grep -v '/api/appid' /var/log/nginx/access.log|awk '{print $7}' | sort |uniq -c | sort -rn | head -n 10

9. View pages with more than 100 page visits

cat /var/log/nginx/access.log | cut -d ' ' -f 7 | sort |uniq -c | awk '{if ($1 > 100) print $0}' | less

10. View the most recent 1,000 records and the most visited pages

tail -1000 /var/log/nginx/access.log |awk '{print $7}'|sort|uniq -c|sort -nr|less

11. Count the number of requests per hour and the time points of the top 10 (accurate to the hour)

awk '{print $4}' /var/log/nginx/access.log |cut -c 14-15|sort|uniq -c|sort -nr|head -n 10

12. Count the number of requests per minute and the time points of the top 10 (accurate to the minute)

awk '{print $4}' /var/log/nginx/access.log |cut -c 14-18|sort|uniq -c|sort -nr|head -n 10

13. Count the number of requests per second and the time points of the top 10 (accurate to seconds)

awk '{print $4}' /var/log/nginx/access.log |cut -c 14-21|sort|uniq -c|sort -nr|head -n 10

14. Find logs for a specified time period

awk '$4 >="[25/Mar/2020:01:00:00" && $4 <="[25/Mar/2020:08:00:00"' /var/log/nginx/access.log

15. List the URLs whose transmission time exceeds 0.6 seconds, and display the first 10

cat /var/log/nginx/access.log |awk '(substr($NF,2,5) > 0.6){print $4,$7,substr($NF,2,5)}' | awk -F '"' '{print $1,$2,$3}' |sort -k3 -rn | head -10

16. List the time points where the request time for /api/appid exceeds 0.6 seconds

cat /var/log/nginx/access.log |awk '(substr($NF,2,5) > 0.6 && $7~/\/api\/appid/){print $4,$7,substr($NF,2,5)}' | awk -F '"' '{print $1,$2,$3}' |sort -k3 -rn | head -10

17. Get the top 10 most time-consuming request times, URLs, and durations

cat /var/log/nginx/access.log |awk '{print $4,$7,substr($NF,2,5)}' | awk -F '"' '{print $1,$2,$3}' | sort -k3 -rn | head -10

Summarize

This is the end of this article about Nginx status monitoring and log analysis. For more relevant Nginx status monitoring and log analysis content, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope you will support 123WORDPRESS.COM in the future!