Docker container monitoring principle and cAdvisor installation and usage instructions

It is very important to monitor the operating status of containers in the production environment. Through monitoring, we can grasp the operating status of the container at any time, so as to discover and solve online risks and problems early.

So today I will share with you the knowledge about container monitoring (principles and tools cAdvisor).

Although traditional physical and virtual machine monitoring already has relatively mature monitoring solutions, container monitoring faces greater challenges because the behavior and nature of containers are different from traditional virtual machines. In general, containers have the following characteristics:

Containers are short-lived and can be dynamically scheduled

The essence of a container is a process, not a complete operating system

Because containers are very lightweight, they are created and destroyed more frequently than traditional virtual machines.

There are many monitoring solutions for Docker containers. In addition to the docker stats command that comes with Docker, there are many open source solutions, such as sysdig, cAdvisor, Prometheus, etc., which are all excellent monitoring tools.

Let's first look at how to use Docker's own docker stats command to monitor containers without the help of any external tools.

Using the docker stats command

Using the docker stats command that comes with Docker, you can easily see the usage of CPU, memory, network IO, disk IO, PID and other resources of all containers on the host. Let’s take a look at the specific operations below.

First, use the following command on the host to start an nginx container with a resource limit of 1 core and 2G:

$ docker run --cpus=1 -m=2g --name=nginx -d nginx

After the container is started, you can use the docker stats command to view the resource usage status of the container:

$ docker stats nginx

Through the docker stats command, you can see the running status of the container as follows:

CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS

f742a467b6d8 0.00% 1.387 MiB / 2 GiB 0.07% 656 B / 656 B 0 B / 9.22 kB 2

From the running status of the container, it can be seen that the docker stats command can indeed obtain and display the running status of the Docker container. However, its shortcomings are also obvious, because it can only obtain local data, cannot view historical monitoring data, and has no visual display panel.

Therefore, we usually use another container monitoring solution cAdvisor in production environments.

cAdvisor

cAdvisor is a general container monitoring solution open sourced by Google. cAdvisor can not only collect information about all containers running on the machine, but also provide a basic query interface and HTTP interface, making it easier to integrate with external systems. Therefore, cAdvisor quickly became the most commonly used component for container metrics monitoring, and Kubernetes also integrated cAdvisor as the default tool for container monitoring metrics.

Installation and use of cAdvisor

Below we take cAdvisor 0.37.0 version as an example to demonstrate the installation and use of cAdvisor.

cAdvisor officially provides a Docker image. We only need to pull the image and start it.

Since the cAdvisor image is stored in Google's gcr.io image repository, it is not accessible in China. Here I put the built image on Docker Hub. You can pull it from Docker Hub using the docker pull lagoudocker/cadvisor:v0.37.0 command.

First start cAdvisor with the following command:

$ docker run \
 --volume=/:/rootfs:ro \
 --volume=/var/run:/var/run:ro \
 --volume=/sys:/sys:ro \
 --volume=/var/lib/docker/:/var/lib/docker:ro \
 --volume=/dev/disk/:/dev/disk:ro \
 --publish=8080:8080 \
 --detach=true \
 --name=cadvisor \
 --privileged \
 --device=/dev/kmsg \
 lagoudocker/cadvisor:v0.37.0

At this point, cAdvisor has been successfully started, and we can access the cAdvisor Web interface by visiting http://localhost:8080.

cAdvisor can monitor not only container resource usage, but also host resource usage. Next, let's take a look at how it checks the host resource usage.

Use cAdvisor to view host monitoring

Visit http://localhost:8080/containers/. You can see the resource usage of the host on the homepage, including CPU, memory, file system, network and other resources, as shown in the figure below.

View container monitoring using cAdvisor

If you want to view the resource usage of containers running on the host, you can visit http://localhost:8080/docker/. This page will list the basic information of Docker and the running containers, as shown in the figure below.

In the above figure, under Subcontainers, all containers running on the current host are listed. Click one of the containers to view the detailed running status of the container, as shown in the following figure.

In general, using cAdvisor to monitor containers has the following features:

Can collect the status of physical machines and containers at the same time

Can display monitoring historical data

Now that you know about Docker's monitoring tools, are you wondering where these monitoring data come from? Next, I will show you the principles of container monitoring.

Monitoring principle

We know that Docker is implemented based on Namespace, Cgroups and Union File System. Cgroups can not only be used to limit container resources, but also provide container resource utilization. Regardless of the implementation of the monitoring solution, the underlying data comes from Cgroups.

The working directory of Cgroups is /sys/fs/cgroup, and the /sys/fs/cgroup directory contains all the contents of Cgroups. Cgroups contains many subsystems that can be used to limit different resources. For example, limit and monitor resources such as CPU, memory, PID, and disk IO.

To understand the Cgroups subsystem in more detail, we use the ls -l command to view the /sys/fs/cgroup folder and see many directories:

$ sudo ls -l /sys/fs/cgroup/
total 0
 
dr-xr-xr-x 5 root root 0 Jul 9 19:32 blkio
lrwxrwxrwx 1 root root 11 Jul 9 19:32 cpu -> cpu,cpuacct
dr-xr-xr-x 5 root root 0 Jul 9 19:32 cpu,cpuacct
lrwxrwxrwx 1 root root 11 Jul 9 19:32 cpuacct -> cpu,cpuacct
dr-xr-xr-x 3 root root 0 Jul 9 19:32 cpuset
dr-xr-xr-x 5 root root 0 Jul 9 19:32 devices
dr-xr-xr-x 3 root root 0 Jul 9 19:32 freezer
dr-xr-xr-x 3 root root 0 Jul 9 19:32 hugetlb
dr-xr-xr-x 5 root root 0 Jul 9 19:32 memory
lrwxrwxrwx 1 root root 16 Jul 9 19:32 net_cls -> net_cls,net_prio
dr-xr-xr-x 3 root root 0 Jul 9 19:32 net_cls,net_prio
lrwxrwxrwx 1 root root 16 Jul 9 19:32 net_prio -> net_cls,net_prio
dr-xr-xr-x 3 root root 0 Jul 9 19:32 perf_event
dr-xr-xr-x 5 root root 0 Jul 9 19:32 pids
dr-xr-xr-x 5 root root 0 Jul 9 19:32 systemd

These directories represent Cgroups subsystems, and Docker will create a docker folder under each Cgroups subsystem. If you don't know much about the Cgroups subsystem, don't worry. You just need to understand that container monitoring data comes from Cgroups.

How does the monitoring system obtain the container's memory limit?

Next, we take the memory subsystem (the memory subsystem is one of the many subsystems of Cgroups, mainly used to limit memory usage) as an example to explain how the monitoring component obtains the resource limits and usage status of the container (that is, the memory limit of the container).

We first start an nginx container with a resource limit of 1 core and 2G on the host using the following command:

$ docker run --name=nginx --cpus=1 -m=2g --name=nginx -d nginx

## The output here is the container ID

51041a74070e9260e82876974762b8c61c5ed0a51832d74fba6711175f89ede1

Note: If you have already created a container named nginx, please use the docker rm -f nginx command to delete the existing nginx container first.

After the container is started, we can get the container ID through the output of the command line. At the same time, Docker will create a corresponding folder in the /sys/fs/cgroup/memory/docker directory with the container ID as the name.

Let's take a look at the files in the /sys/fs/cgroup/memory/docker directory:

$ sudo ls -l /sys/fs/cgroup/memory/docker
total 0
 
drwxr-xr-x 2 root root 0 Sep 2 15:12 51041a74070e9260e82876974762b8c61c5ed0a51832d74fba6711175f89ede1
-rw-r--r-- 1 root root 0 Sep 2 14:57 cgroup.clone_children
--w--w--w- 1 root root 0 Sep 2 14:57 cgroup.event_control
-rw-r--r-- 1 root root 0 Sep 2 14:57 cgroup.procs
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.failcnt
--w------ 1 root root 0 Sep 2 14:57 memory.force_empty
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.kmem.failcnt
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.kmem.limit_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.kmem.max_usage_in_bytes
-r--r--r-- 1 root root 0 Sep 2 14:57 memory.kmem.slabinfo
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.kmem.tcp.failcnt
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.kmem.tcp.limit_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.kmem.tcp.max_usage_in_bytes
-r--r--r-- 1 root root 0 Sep 2 14:57 memory.kmem.tcp.usage_in_bytes
-r--r--r-- 1 root root 0 Sep 2 14:57 memory.kmem.usage_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.limit_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.max_usage_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.memsw.failcnt
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.memsw.limit_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.memsw.max_usage_in_bytes
-r--r--r-- 1 root root 0 Sep 2 14:57 memory.memsw.usage_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.move_charge_at_immigrate
-r--r--r-- 1 root root 0 Sep 2 14:57 memory.numa_stat
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.oom_control
---------- 1 root root 0 Sep 2 14:57 memory.pressure_level
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.soft_limit_in_bytes
-r--r--r-- 1 root root 0 Sep 2 14:57 memory.stat
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.swappiness
-r--r--r-- 1 root root 0 Sep 2 14:57 memory.usage_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 14:57 memory.use_hierarchy
-rw-r--r-- 1 root root 0 Sep 2 14:57 notify_on_release
-rw-r--r-- 1 root root 0 Sep 2 14:57 tasks

You can see that Docker has created a directory named after the container ID. Let's use the ls command to view the contents of the directory:

$ sudo ls -l /sys/fs/cgroup/memory/docker/51041a74070e9260e82876974762b8c61c5ed0a51832d74fba6711175f89ede1
 
total 0
-rw-r--r-- 1 root root 0 Sep 2 15:21 cgroup.clone_children
--w--w--w- 1 root root 0 Sep 2 15:13 cgroup.event_control
-rw-r--r-- 1 root root 0 Sep 2 15:12 cgroup.procs
-rw-r--r-- 1 root root 0 Sep 2 15:12 memory.failcnt
--w------ 1 root root 0 Sep 2 15:21 memory.force_empty
-rw-r--r-- 1 root root 0 Sep 2 15:21 memory.kmem.failcnt
-rw-r--r-- 1 root root 0 Sep 2 15:12 memory.kmem.limit_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 15:21 memory.kmem.max_usage_in_bytes
-r--r--r-- 1 root root 0 Sep 2 15:21 memory.kmem.slabinfo
-rw-r--r-- 1 root root 0 Sep 2 15:21 memory.kmem.tcp.failcnt
-rw-r--r-- 1 root root 0 Sep 2 15:21 memory.kmem.tcp.limit_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 15:21 memory.kmem.tcp.max_usage_in_bytes
-r--r--r-- 1 root root 0 Sep 2 15:21 memory.kmem.tcp.usage_in_bytes
-r--r--r-- 1 root root 0 Sep 2 15:21 memory.kmem.usage_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 15:12 memory.limit_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 15:12 memory.max_usage_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 15:21 memory.memsw.failcnt
-rw-r--r-- 1 root root 0 Sep 2 15:12 memory.memsw.limit_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 15:21 memory.memsw.max_usage_in_bytes
-r--r--r-- 1 root root 0 Sep 2 15:21 memory.memsw.usage_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 15:21 memory.move_charge_at_immigrate
-r--r--r-- 1 root root 0 Sep 2 15:21 memory.numa_stat
-rw-r--r-- 1 root root 0 Sep 2 15:13 memory.oom_control
---------- 1 root root 0 Sep 2 15:21 memory.pressure_level
-rw-r--r-- 1 root root 0 Sep 2 15:21 memory.soft_limit_in_bytes
-r--r--r-- 1 root root 0 Sep 2 15:21 memory.stat
-rw-r--r-- 1 root root 0 Sep 2 15:21 memory.swappiness
-r--r--r-- 1 root root 0 Sep 2 15:12 memory.usage_in_bytes
-rw-r--r-- 1 root root 0 Sep 2 15:21 memory.use_hierarchy
-rw-r--r-- 1 root root 0 Sep 2 15:21 notify_on_release
-rw-r--r-- 1 root root 0 Sep 2 15:21 tasks

As can be seen above, there are many files in the directory of the container ID. The memory.limit_in_bytes file represents the memory limit size of the container in bytes. We use the cat command (the cat command can view the file content) to view the file content:

$ sudo cat /sys/fs/cgroup/memory/docker/51041a74070e9260e82876974762b8c61c5ed0a51832d74fba6711175f89ede1/memory.limit_in_bytes

2147483648

Here you can see that the value of memory.limit_in_bytes is 2147483648, which is exactly 2G after conversion, which is in line with the memory limit of 2G when we started the container.

Through the example of the memory subsystem, we can know that the monitoring component can obtain the container memory limit value by reading the memory.limit_in_bytes file. After understanding the memory limit of the container, let's take a look at the memory usage of the container.

$ sudo /sys/fs/cgroup/memory/docker/51041a74070e9260e82876974762b8c61c5ed0a51832d74fba6711175f89ede1/memory.usage_in_bytes

4259840

You can see that the current memory usage is 4259840 bytes, about 4 MB. Understand memory monitoring.

Next, let's take a look at the sources of network monitoring data.

The network monitoring data source is read from the /proc/{PID}/net/dev directory, where PID is the process ID of the container on the host. Next, we first use the docker inspect command to view the PID of the nginx container started above. The command is as follows:

$ docker inspect nginx |grep Pid
 
   "Pid": 27348,
 
   "PidMode": "",
 
   "PidsLimit": 0,

You can see that the PID of the container is 27348. Use the cat command to view the contents of /proc/27348/net/dev.

$ sudo cat /proc/27348/net/dev
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
 
 lo: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 
 eth0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

The /proc/27348/net/dev file records the traffic received and sent by each network card in the container, as well as the number of errors, number of packet losses, and other information. It can be seen that the network monitoring data of the container is read and displayed from here regularly.

To sum up, the monitoring principle of the container is actually to periodically read the relevant files on the Linux host and display them to the user.

Conclusion

k8s uses metrics serve, cAdvisor provides the underlying data, and the underlying data source of metrics-server is cAdvisor

cAdvisor provides monitoring data, and Prometheus is responsible for collecting data. These two functions are different. In production clusters, cAdvisor is generally used together with Prometheus.

The above article on Docker container monitoring principles and the installation and use of cAdvisor is all the content that the editor shares with you. I hope it can give you a reference. I also hope that you will support 123WORDPRESS.COM.