Detailed explanation of the process of building Prometheus+Grafana based on docker

1. Introduction to Prometheus

Prometheus is an open source monitoring, alarm, and time series database combination, originally developed by SoundCloud. As it develops, more and more companies and organizations are willing to adopt Prometheus, and the community is very active. They have separated it into an open source project and have a company operate it. Google SRE's book also mentioned that an implementation similar to their BorgMon monitoring system is Prometheus. The most common Kubernetes container management system is usually monitored with Prometheus.

The basic principle of Prometheus is to periodically capture the status of monitored components through the HTTP protocol. The advantage of this is that any component can be connected to the monitoring system as long as it provides an HTTP interface, without the need for any SDK or other integration process. This is very suitable for virtualized environments such as VM or Docker.

Prometheus should be one of the few monitoring systems suitable for Docker, Mesos, and Kubernetes environments.

The HTTP interface that exports information about monitored components is called exporter. Currently, most of the components commonly used by Internet companies have exporters that can be used directly, such as Varnish, Haproxy, Nginx, MySQL, Linux system information (including disk, memory, CPU, network, etc.). For specific supported sources, see: https://github.com/prometheus.

Compared with other monitoring systems, the main features of Prometheus are:

• A multi-dimensional data model (time series are defined by metric names and key/value dimensions).
• Very efficient storage. On average, one sampled data occupies about 3.5 bytes. 3.2 million time series are sampled every 30 seconds and kept for 60 days, consuming about 228G of disk space.
• A flexible query language.
• Does not rely on distributed storage, a single server node.
• Event collection is done via a PULL model over HTTP.
• Support push time through intermediate gateways.
• Discover targets via service discovery or static configuration.
• Multiple modes of graphics and dashboard support.

2. Overview of Prometheus Architecture

This diagram illustrates the overall architecture of Prometheus and some of its ecosystem components:

Its service process is as follows: Prometheus daemon is responsible for periodically fetching metrics data from the target. Each fetch target needs to expose an http service interface for periodic fetching.

Prometheus: supports specifying crawling targets through configuration files, text files, zookeeper, Consul, DNS SRV lookup, etc. It supports many types of chart visualization, such as the very exquisite Grafana, the built-in Promdash, and its own template engine, etc. It also provides HTTP API query methods to customize the required output.

Alertmanager: It is a component independent of Prometheus, which can support Prometheus query statements and provide a very flexible alarm method.

PushGateway: This component supports the Client to actively push metrics to PushGateway, while Prometheus only periodically fetches data from the Gateway.

If you have used statsd, you will find it very similar, except that statsd sends data directly to the server, while Prometheus mainly relies on the process to actively capture.

Most Prometheus components are written in Go, and they can be easily built and deployed as static binaries. Visit prometheus.io for complete documentation, examples, and guides.

3. Prometheus data model

Fundamentally, all storage in Prometheus is implemented in time series. The same metrics (metric name) and label (one or more labels) form a time series, and different labels represent different time series. In order to support some queries, some time series storage is sometimes temporarily generated.

metrics name&label indicator name and label

Each time series consists of a unique "metric name" and a set of "labels (key=value)".

Indicator name: Generally, a name is given to the monitoring object, such as http_requests_total. It has some naming rules and can contain letters, numbers, etc. It usually starts with application name_monitoring object_value type_unit. For example: push_total, userlogin_mysql_duration_seconds, app_memory_usage_bytes.

Tags: are used to identify different dimensions of a time series. For example, whether an http request uses POST or GET, and what its endpoint is, tags are needed to mark it. The final identifier is: http_requests_total{method="POST",endpoint="/api/tracks"}.

Remember, adding or removing a tag for the metric name http_requests_total will create a new time series.

The query statement can query the aggregation results based on the combination of the above tags.

If we look at this statement from the perspective of a traditional database, we can consider http_requests_total to be the table name, the tag to be the field, timestamp to be the primary key, and a float64 field to be the value. (All values ​​in Prometheus are stored as float64).

4. Four data types of Prometheus

Counter

Counter is used to accumulate values, such as the number of requests, the number of completed tasks, and the number of errors. It keeps increasing and never decreases. After restarting the process, it will be reset.

For example: http_response_total{method=”GET”,endpoint=”/api/tracks”} 100, crawl http_response_total{method=”GET”,endpoint=”/api/tracks”} 100 after 10 seconds.

Gauge

Gauge general values, such as temperature changes and memory usage changes. It can be made bigger or smaller. After restarting the process, it will be reset.

For example: memory_usage_bytes{host="master-01"} 100 < the captured value, memory_usage_bytes{host="master-01"} 30, memory_usage_bytes{host="master-01"} 50, memory_usage_bytes{host="master-01"} 80 < the captured value.

Histogram

Histogram can be understood as a bar graph, which is often used to track the scale of events, such as request duration and response size. What makes it special is that it can group the contents of the records and provide the functions of count and sum of all values.

For example: {less than 10=5 times, less than 20=1 time, less than 30=2 times}, count=7 times, sum=the sum of 7 times.

Summary

Summary is very similar to Histogram and is often used to track the scale of events, such as request duration and response size. It also provides functions for counting and summing all values.

For example: count = 7 times, sum = 7 times of value evaluation.

It provides a quantiles function that can divide the tracking results by % ratio. For example, a quantile value of 0.95 means that 95% of the data in the sample value is taken.

5. Install and run Prometheus (docker version)

The following describes how to use Prometheus and Grafana to monitor the performance of the local server.

To monitor the local machine, only one exporter is needed

node_exporter – for machine system data collection

Grafana is an open source, feature-rich data visualization platform that is often used for visualizing time series data. It has built-in support for the following data sources:

The following is the architecture diagram we used during installation:

Note: This article uses ubuntu-16.04.5-server-amd64, only one server is needed!

Install Docker

apt-get install -y docker.io

Note: Articles online say that you need to install docker-engine and docker-ce, which is nonsense. The bag can't be found at all!

Just install docker.io and you’re done!

If it is a CentOS system, use yum install -y docker-io to install

Download the image package

docker pull prom/node-exporter
docker pull prom/prometheus
docker pull grafana/grafana

Start node-exporter

docker run -d -p 9100:9100 \
 -v "/proc:/host/proc:ro" \
 -v "/sys:/host/sys:ro" \
 -v "/:/rootfs:ro" \
 --net="host" \
 prom/node-exporter

Wait a few seconds to see if the port is up.

root@ubuntu:~# netstat -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1147/sshd    
tcp 0 36 192.168.91.132:22 192.168.91.1:63648 ESTABLISHED 2969/0     
tcp 0 0 192.168.91.132:22 192.168.91.1:63340 ESTABLISHED 1321/1     
tcp6 0 0 :::9100 :::* LISTEN 3070/node_exporter

Visit url:

http://192.168.91.132:9100/metrics

The effect is as follows:

These are all collected data, with it you can do data display

Start prometheus

Create a new directory prometheus and edit the configuration file prometheus.yml

mkdir /opt/prometheus
cd /opt/prometheus/
vim prometheus.yml

The content is as follows:

global:
 scrape_interval: 60s
 evaluation_interval: 60s
 
scrape_configs:
 - job_name: prometheus
  static_configs:
   - targets: ['localhost:9090']
    labels:
     instance: prometheus
 
 - job_name: linux
  static_configs:
   - targets: ['192.168.91.132:9100']
    labels:
     instance: localhost

Note: Change the IP address. Here 192.168.91.132 is the local address.

Start prometheus

docker run -d \
 -p 9090:9090 \
 -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
 prom/prometheus

Wait a few seconds and check the port status

root@ubuntu:/opt/prometheus# netstat -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1147/sshd    
tcp 0 36 192.168.91.132:22 192.168.91.1:63648 ESTABLISHED 2969/0     
tcp 0 0 192.168.91.132:22 192.168.91.1:63340 ESTABLISHED 1321/1     
tcp6 0 0 :::9100 :::* LISTEN 3070/node_exporter
tcp6 0 0 :::22 :::* LISTEN 1147/sshd    
tcp6 0 0 :::9090 :::* LISTEN 3336/docker-proxy

Visit url:

http://192.168.91.132:9090/graph

The effect is as follows:

Access targets, the url is as follows:

http://192.168.91.132:9090/targets

The effect is as follows:

If the status is not UP, wait for a while and it will be UP

Start Grafana

Create a new empty folder grafana-storage to store data

mkdir /opt/grafana-storage

Set permissions

chmod 777 -R /opt/grafana-storage

Because the grafana user will write files in this directory, it is relatively simple and crude to set 777 directly!

Start Grafana

docker run -d \
 -p 3000:3000 \
 --name=grafana \
 -v /opt/grafana-storage:/var/lib/grafana \
 grafana/grafana

Wait a few seconds and check the port status

root@ubuntu:/opt/prometheus# netstat -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1147/sshd    
tcp 0 36 192.168.91.132:22 192.168.91.1:63648 ESTABLISHED 2969/0     
tcp 0 0 192.168.91.132:22 192.168.91.1:63340 ESTABLISHED 1321/1     
tcp6 0 0 :::9100 :::* LISTEN 3070/node_exporter
tcp6 0 0 :::22 :::* LISTEN 1147/sshd    
tcp6 0 0 :::3000 :::* LISTEN 3494/docker-proxy
tcp6 0 0 :::9090 :::* LISTEN 3336/docker-proxy
tcp6 0 0 192.168.91.132:9100 172.17.0.2:55108 ESTABLISHED 3070/node_exporter

Visit url:

http://192.168.91.132:3000/

By default, you will be redirected to the login page. The default username and password are both admin.

After logging in, it will ask you to reset your password. You can enter the admin password again!

After the password is set, you will be redirected to the home page.

Click Add data source. Since the mirroring method is used, the version is relatively new. It’s different from the pictures shown in the articles on the Internet!

name: Prometheus

Type selects Prometheus, because the data is obtained from it

URL Enter the IP+port of Prometheus

Click Save & Test below. If a green icon appears, it means it is OK.

Go back to the home page and click New dashboard

Click Graph

The effect is as follows:

Click Edit below the title.

The effect is as follows:

Enter cpu, there will be a prompt at the bottom

Here we monitor node_load15, which indicates the system load for 15 minutes. Click Add Query below

The effect is as follows:

Add Total Memory

There will be one more line here

Click on the right to delete the total memory

Click General and change the title to Chinese

The chart effect is as follows:

Click the Save button above

Enter your name

The effect is as follows:

Click on the homepage and you will see

Reference links for this article:

http://www.ywnds.com/?p=9656

This is the end of this article about the detailed process of building Prometheus+Grafana based on docker. For more relevant content about building Prometheus+Grafana with docker, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope you will support 123WORDPRESS.COM in the future!