Detailed method of using goaccess to analyze nginx logs

Recently I want to use goaccess to analyze nginx logs, but the configuration format of nginx logs is not in the normal format. It is completely written according to our own needs, so goaccess cannot analyze it and we need to redefine the format ourselves. Although there are many introductions to goaccess on the Internet, most of them focus on the important points and ignore the customization of the format, so I will talk about the customization.

GoAccess is an open source, real-time web log analysis tool that runs in a command line terminal. This tool provides fast and diverse HTTP status statistics, allowing administrators to stop worrying about counting various types of data and say goodbye to complicated commands and a lot of pipelines/regular expressions.

Analyze nginx logs

GoAccess's various display modes
Goaccess has multiple ways of data visualization, namely:

Output formatted data from the command line Use access.log to generate static visualization data Generate real-time visualization data Note that if you compile and install and select --enable-geoip=mmdb, you need to edit the configuration file and bring the parameter --config-file=/usr/local/etc/goaccess/goaccess.conf when using the command. This is not required if you install using a package manager

Command line output GoAccess
goaccess /var/log/nginx/access.log -c, it will first ask you the format of the data. I use the first format for the log here.

Parse accesslog to generate static html
GoAccess can also parse access.log to generate static HTML to display data in a more intuitive way.

goaccess /var/log/nginx/access.log -o report.html --log-format=COMBINED, then use the browser to access report.html to view the report, which contains all the data.

Real-time parsing of access logs
In addition to generating static HTML files, GoAccess can also generate real-time website access data!

goaccess /var/log/nginx/access.log -o /var/www/html/report.html --log-format=COMBINED --real-time-html --config-file=/usr/local/etc/goaccess/goaccess.conf

Add Chinese support
Goaccess 1.3 and later versions provide multi-language support. First, execute apt install language-pack-zh-hans in the command line to install the Chinese package, then use export LANG=zh_CN.UTF-8 to modify the environment variable, and then use goaccess /var/log/nginx/access.log -o /var/www/html/report.html --log-format=COMBINED --real-time-html --config-file=/usr/local/etc/goaccess/goaccess.conf to start GoAccess again, and you can find that the interface is already in Chinese.

For real-time mode, you can check the demo on the official website https://rt.goaccess.io/?20200209201008

Abnormal exit If the real-time mode is not exited normally, it may not be able to start normally again. GoAccess uses the 7890 websocket port by default, so use lsof -i:7890 to view the process number occupying the port and kill it.

SSL support If you need to output real-time data on an encrypted connection, you need to use --ssl-cert= and --ssl-key=. After setting it up, I visited report.html and found that the data was still static. Suddenly I remembered that I used cloudflare cdn, and port 7890 was not in cloudflare's supported port list, so I used the parameters --ws-url=wss://server domain name (our browser will try to connect to port 8443 of the domain name):8443 --port=8443 to change the port to 8443. What is unexpected is that at this time, report.html can be connected when using the proxy link, and real-time information can be viewed, but when connected directly, it is still static data, as shown by tcping.

Go to cloudflare's official website and you can find the following content

Only ports 80 and 443 are compatible with the following services:

For HTTP/HTTPS traffic from data centers in China with domain names enabled for China Network,
In other words, it is impossible to connect to non-80/443 ports through Cloudflare in China...

Reverse proxy But it's not that there is no way to connect. Finally, I thought of the reverse proxy solution.

Change the startup parameters to --ws-url=wss://yourdomain.com/goaccess --port=7890

Modify the nginx site configuration file /etc/nginx/site-available/default and add the following content

location /goaccess {
    proxy_redirect off;
    proxy_pass https://127.0.0.1:7890;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
}

Note that if URL rewriting is enabled in your site configuration file, in order to avoid /goaccess being affected, we need to exclude this path from being rewritten.

Put all rewrite rules in location /

location / { 
    if (-f $request_filename/index.html){
    rewrite (.*) $1/index.html break;
    }
    if (-f $request_filename/index.php){
    rewrite (.*) $1/index.php;
    }
    if (!-f $request_filename){
    rewrite (.*) /index.php;
    }
}

You don't need to do anything below

location /goaccess/ {
}

After that, restart nginx and access report.html again. You will find that the gear on the left finally shows connect.

If you are just watching it yourself or don't care about the IP being exposed, it would be less troublesome to connect directly using the IP instead of going through the CDN.

This is the end of this article about using goaccess to analyze nginx logs. For more related goaccess analysis of nginx log content, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope you will support 123WORDPRESS.COM in the future!