How to collect Nginx logs using Filebeat

Nginx logs can be used to analyze user address locations, behavior profiles, etc. How can we use Elastic Stack to perform one-stop data collection, data cleaning, data landing, and data visualization to make the data truly valuable?

Architecture Design

In the Elastic Stack, Filebeat is used to collect Nginx-related logs, Elasticsearch is an engine for data storage and search, and Kibana is a tool for data visualization.

In Nginx, the relevant logs are stored in the /var/log/nginx directory, namely the access log access.log and the error log error.log.

If it is a bare metal environment, you can directly install Filebeat on the same host to collect log files.
If it is a Docker environment, it is recommended that Nginx use Volume to share log files for Filebeat collection.
If it is a Kubernetes environment, it is recommended to add Filebeat Container to the Pod to collect PV.

There are different collection solutions for different scenarios. Some can use Daemonset to collect logs on the host, while others can use Sidecar to collect logs, depending on the business scenario.

Implementation Methods

Take Docker environment as an example

Nginx

Create a storage volume to facilitate the joint mounting of Nginx and Filebeat containers
docker volume create nginx-log-volume

Start the Nginx container and map the storage volume to the log directory
docker run -d --name nginx -p 80:80 -v nginx-log-volume:/var/log/nginx nginx:latest

Enter the container to modify the configuration
docker exec -it nginx /bin/bash

Since the default log in the container environment is output to stdout, cancel this setting and specify a file
unlink /var/log/nginx/access.log
unlink /var/log/nginx/error.log
touch /var/log/nginx/access.log /var/log/nginx/error.log
nginx -s reload

Filebeat

Start the Filebeat container and map the storage volume to the data directory
docker run -d --name filebeat --user=root -v nginx-log-volume:/data elastic/filebeat:7.9.2

Enter the container to modify the configuration
docker exec -it filebeat /bin/bash

Modify the configuration and add the hosts for Elasticsearch and Kibana
vi filebeat.yml

filebeat.config:
 modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

processors:
 - add_cloud_metadata: ~
 - add_docker_metadata: ~

output.elasticsearch:
 hosts: 'elasticsearch:9200'
 username: "elastic"
 password: "xxx"
setup.kibana:
 host: "kibana:5601"

Enable Nginx collection module

filebeat modules enable nginx

Edit Nginx collection configuration
vi modules.d/nginx.yml

- module: nginx
 access:
  enabled: true
  var.paths: ["/data/access.log*"]
 error:
  enabled: true
  var.paths: ["/data/error.log*"]

Set up Filebeat to create an Index Pattern and Dashboard on Kibana
filebeat setup

Restart Filebeat to take effect
docker restart filebeat

Visualization

Use the Dashboard function in Kibana to display Nginx's access to logs, user address location, and browser information

Displays Nginx's specific request information for access logs and error logs