Example of using Docker to build an ELK log system

The following installations all use the ~/ directory as the installation root directory.

ElasticSearch

Download image:

$ sudo docker pull elasticsearch:5.5.0

Run the ElasticSearch container:

$ sudo docker run -it -d -p 9200:9200 -p 9300:9300 \
-v ~/elasticsearch/data:/usr/share/elasticsearch/data \
--name myes elasticsearch:5.5.0

Please note that if you use version 6 or above, JDK errors will occur. Check the logs.

$ docker logs -f myes

View the logs:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

The approximate meaning found on the Internet is:

jdk9 is not very friendly to elasticSearch (the version is too new), and JDK8 must be used. I use JDK8u152 (jdk-8u152-windows-x64.exe). If you use JDK9 and elasticSearch-rtf (v5.1.1), the following error will occur. Please pay special attention that the elasticSearch6.0 version must use JDK9, otherwise the msi downloaded from the official website cannot be installed successfully. The reason has not been carefully checked.

So this is also a very annoying problem, so I just installed the v5.5.0 stable version directly.

Logstash

Download image:

$ sudo docker pull logstash:5.5.0

Create a new configuration file:

$ mkdir ~/logstash/conf.d && cd logstash/conf.d
$ vim logstash.conf

logstash.conf:

input {

 beats {
  port => 5044 # This port needs to be the same as the port in filebeat.yml}

 file {
  path => "/data/logs"
  # start_position => "beginning"
 }
}

filter {
 #grok {
 # match => { "message" => "%{COMBINEDAPACHELOG}" }
 #}
 #date {
 # match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
 #}

 grok {

  patterns_dir => "/etc/logstash/conf.d/patterns"
  match => {"message" => "%{TIMESTAMP_ISO8601:time}\S%{MYSELF:msgid}%{MYSELF:method}%{MYDATA:data}%{MYSELF:UserInfo}\S%{LOGLEVEL:level}\S%{MYSELF:thread}%{MYSELF:application}%{MYSELF:ip}"}
}
date {
   #match => [ "time", "YYYY-MM-dd HH:mm:ss,SSS" ]
   match => [ "time", "ISO8601" ]
   target => "@timestamp"
   timezone => "Asia/Phnom_Penh"
 }

}

output {

 stdout {
  codec => rubydebug
 }

 elasticsearch
  action => "index"
  hosts => ["172.17.10.114:9200"]
  index => "%{[fields][indexname]}-%{+YYYY.MM.dd}"
 }
}

Run the Logstash container:

$ sudo docker run -it -d -p 5044:5044 \
-v ~/logstash/conf.d:/etc/logstash/conf.d \
-v ~/logstash/data/logs:/data/logs \
--name logstash logstash:5.5.0 \
-f /etc/logstash/conf.d/logstash.conf

Kibana

Download image:

$ sudo docker pull kibana:5.5.0

Create a new configuration file:

$ mkdir ~/kibana && cd ~/kibana
$ vim kibana.yml

kibana.yml:

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://172.17.10.114:9200"

Run the Kibana container:

$ sudo docker run -it -d -p 5601:5601 \
-v ~/kibana:/etc/kibana \
--name kibana kibana:5.5.0

Filebeat

Filebeat needs to be deployed on the server where logs need to be collected.

Download image:

$ sudo docker pull docker.elastic.co/beats/filebeat:5.5.0

Create a new configuration file:

filebeat.prospectors:
- type: log
  paths:
   - ~/filebeat/logs #Specify the path of the log files to be collected fields:
  indexname: xxx # Fill in the project name here, corresponding to index => "%{[fields][indexname]}-%{+YYYY.MM.dd}"
output.logstash:
 hosts: ["172.17.10.114:5044"]

Run the Filebeat container:

$ sudo docker run -it -d \
-v ~/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml \
--name filebeat docker.elastic.co/beats/filebeat:5.5.0

Attached is a ELK structure flow chart:

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.

You may also be interested in: