How to quickly build ELK based on Docker

[Abstract] This article quickly builds a complete ELK system based on the self-built Docker platform. The relevant images are directly obtained from the Docker Hub, which can quickly realize log collection, analysis and retrieval.

Preparing the image

• Get the ES image: docker pull elasticsearch:latest
• Get the kibana image: docker pull kibana:latest
• Get the logstash image: docker pull logstash:latest

Start Elasticsearch

The ES configuration file in the official image is saved in /usr/share/elasticsearch/config. If necessary, this directory can be mapped to the host machine; the data file directory is /usr/share/elasticsearch/data. Here we map the data directory out; the container provides port 9200 by default for API interaction.

docker run --name elasticsearch \
-v "$PWD/esdata":/usr/share/elasticsearch/data \
-p 9200:9200 \
-d elasticsearch

After the container is started, you can call it to verify:

Start Kibana

Kibana, as the UI for ES operations, needs to communicate with the ES container, so here we need to link the ES container and provide port 5601 for page interaction.

docker run --name kibana \
--link elasticsearch:elasticsearch \
-p 5601:5601 \
-d kibana

After the container is started, use a browser to access port 5601 to see the kibana page. When you access it for the first time, you may be prompted that no default index has been created. Here you need to create a default index on the management page. The default index is usually called logstash-*. Create a default index as shown in the following figure.

Start Logstash

The main function of Logstash is to collect logs. This component has many plug-ins and can support most log integration methods, such as TCP, UDP, JDBC, files, queues, etc. Its configuration is very simple and its startup method is also very simple. Here, taking the access log of nginx as an example, we configure logstash to read the access.log of nginx and then forward the log to Elasticsearch.

First compile a logstash configuration file logstash.conf, the content is as follows:

input{
 
   file{
 
      path=>"/tmp/nginx/logs/access.log"
 
   }
 
}output{
 
   stdout{ } #log output to console #output to es
 
   elasticsearch
 
      hosts=>"100.100.x.231"
 
   }
 
}

Start the container. Here we put the nginx log in /tmp/nginx/logs/access.log. In order for the container to read this log, you need to map the log directory to the container.

docker run –it –rm -v /tmp/nginx/logs/access.log:/tmp/nginx/logs/access.log -f /config-dir/logstash.conf

Next, we can test the entire process of log collection and display. First, create some access logs in nginx, for example, directly use curl to call the nginx service port, or directly write data to access.log. At this time, we can see the following log output in the logstash container:

Later, open the Kibana page to see the log data written in real time:

Summarize

Docker containers make it very convenient to build ELK. Through ELK, you can quickly analyze and retrieve logs and find problems. Several core contributors of ELK founded a company called Elastic, which currently has some cooperation with our company. Based on open source, the company has also released some commercial products called X-Pack, which provides many enhancements in machine learning, graph algorithms, and security technologies. Interested students can learn about it for themselves.

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.