Docker container log analysis

View container logs

First, use docker run -it --rm -d -p 80:80 nginx:1.15.8-alpine command to start an nginx container. If there is no exception, you will get a long string of container IDs such as d2408a7931c95a3a83ffeca2fba887763cf925a67890ef3be4d9ff838aa25b00 . Then use curl -i http://127.0.0.1 to access the service and confirm that the nginx container is started and running normally. Finally, use docker logs -f d24 ​​to view the log output of the container, which is as follows:

172.17.0.1 - - [24/Mar/2019:03:51:21 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"

Generally speaking, the first 3 digits of the container ID are sufficient.

The above is our daily method of viewing container logs, which is very simple and practical.

Container log file storage

The container logs are stored in the local disk as json files. You can use the following method to view the file path: docker inspect d42 | grep Log to find:

"LogPath": "/var/lib/docker/containers/d2408a7931c95a3a83ffeca2fba887763cf925a67890ef3be4d9ff838aa25b00/d2408a7931c95a3a83ffeca2fba887763cf925a67890ef3be4d9ff838aa25b00-json.log",

Note 1: There is no /var/lib/docker directory on Mac, because Docker for Mac works differently, it is best to practice using a Linux system.

Note 2: If the LogPath content is empty, it is probably because of the docker engine version. You can upgrade the docker version to docker-ce 18.09.3

Check the d2408a7931c95a3a83ffeca2fba887763cf925a67890ef3be4d9ff838aa25b00-json.log file and you can see:

{"log":"172.17.0.1 - - [24/Mar/2019:03:51:21 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.29.0\" \"-\"\r\n","stream":"stdout","time":"2019-03-24T03:51:21.982476951Z"}

The content of the log field of this information is consistent with the content previously viewed through the docker logs command.

Container logs follow the container lifecycle and are destroyed when the container is destroyed. Use docker stop 24 to shut down the tested nginx service. Because the --rm parameter is used when the container is started, it will be automatically cleaned up and deleted after shutdown, so you will find that the /var/lib/docker/containers/d2408a7931c95a3a83ffeca2fba887763cf925a67890ef3be4d9ff838aa25b00 directory no longer exists, and the corresponding log files are also deleted.

Container log file rotation strategy

By default, Docker container logs are written to JSON files, which may cause the disk to be full when running online. You can adjust the strategy and let it roll. Modify /etc/docker/daemon.json (if it does not exist, create one manually) and add the following content:

{
 "log-opts": {
 "max-size": "1m",
 "max-file": "3"
 }
}

After the modification is complete, restart the Docker service:

systemctl daemon-reload
systemctl restart docker.service

To test the new logging policy, create a container using the following command:

docker run -d --rm alpine:3.6 sh -c "while true; do echo hello world; usleep 10; done"

This alpine container outputs hello world every 10 microseconds, maintains high-frequency output, and quickly produces log files.

Note: Time control in shell

1. sleep: The default value is seconds.
sleep 1s means delay for one second
sleep 1m means delay for one minute
sleep 1h means delay for one hour
sleep 1d means delay for one day

2. usleep: The default is in microseconds.

1s = 1000ms = 1000000us

Follow the method of viewing the log file in the previous article

# pwd
/var/lib/docker/containers/aa3307f5b42770319129e126122be123cfd8e0ebe1c412371ad27e62faa007e3
# ls -lah
total 2.6M
drwx------ 4 root root 4.0K Mar 24 16:22 .
drwx------ 3 root root 4.0K Mar 24 16:21 ..
-rw-r----- 1 root root 647K Mar 24 16:22 aa3307f5b42770319129e126122be123cfd8e0ebe1c412371ad27e62faa007e3-json.log
-rw-r----- 1 root root 977K Mar 24 16:22 aa3307f5b42770319129e126122be123cfd8e0ebe1c412371ad27e62faa007e3-json.log.1
-rw-r----- 1 root root 977K Mar 24 16:21 aa3307f5b42770319129e126122be123cfd8e0ebe1c412371ad27e62faa007e3-json.log.2

It is easy to find that the log file strategy is to maintain three 1M files, which is consistent with our settings.

After the test is completed, remember to use docker stop aa3 to clean up the test site. The max-size can also be adjusted according to actual needs.

nginx container logs

After understanding the log strategy of Docker containers, let's take a look at how commonly used containers are handled. Let’s take a look at the nginx container first.

First, docker run -it --rm -d -p 80:80 nginx:1.15.8-alpine creates an nginx container, then docker exec -it b6d sh enters the container and checks /etc/nginx/nginx.conf to see the following:

error_log /var/log/nginx/error.log warn;
access_log /var/log/nginx/access.log main;

That is, nginx will write error logs and access logs to the corresponding log files. Continue to view the /var/log/nginx directory:

/var/log/nginx # ls -lah
total 0
drwxr-xr-x 2 root root 39 Mar 4 07:54 .
drwxr-xr-x 3 root root 18 Mar 4 07:54 ..
lrwxrwxrwx 1 root root 11 Jan 31 23:32 access.log -> /dev/stdout
lrwxrwxrwx 1 root root 11 Jan 31 23:32 error.log -> /dev/stderr

This is where the secret comes in. The access.log file is redirected to the standard output through a soft link, while the error log error.log redirects the standard error. In this way, you can use the docker log command to see the nginx access log.

For further verification, check the nginx dockerfile file, which contains:

# forward request and error logs to docker log collector
 && ln -sf /dev/stdout /var/log/nginx/access.log \
 && ln -sf /dev/stderr /var/log/nginx/error.log

It can be seen that the output of the log file is defined when the nginx image is created.

Also use docker stop 524 to clean up the site. This step of cleaning up will not be introduced in the future.

mysql container log

Start a mysql container

docker run --rm -e MYSQL_ROOT_PASSWORD=123456 mysql:5.7

It is not difficult to see the mysql container log output, and the following is a snippet:

Initializing database
2019-03-24T08:48:19.102726Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-03-24T08:48:20.241459Z 0 [Warning] InnoDB: New log files created, LSN=45790
2019-03-24T08:48:20.414933Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2019-03-24T08:48:20.509897Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 935a6ee7-4e11-11e9-b135-0242ac110002.
2019-03-24T08:48:20.519148Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2019-03-24T08:48:20.519843Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2019-03-24T08:48:24.066683Z 1 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2019-03-24T08:48:24.066730Z 1 [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-03-24T08:48:24.066740Z 1 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-03-24T08:48:24.066756Z 1 [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-03-24T08:48:24.066761Z 1 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-03-24T08:48:24.066772Z 1 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2019-03-24T08:48:24.066814Z 1 [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-03-24T08:48:24.066822Z 1 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
Database initialized
Initializing certificates
Generating a RSA private key

Looking at the MySQL Dockerfile file, we can see that the MySQL image startup entry is in entrypoint.sh. From the script, we can find:

echo 'Initializing database'
"$@" --initialize-insecure
echo 'Database initialized'

This is how the output when the mysql container is started corresponds. entrypoint.sh is relatively complex. Its main function is to start mysqld and output logs. Since it is not the focus of this article, it will not be introduced in detail.

Summarize

1. By default, the Docker container outputs to a local JSON file, and its size and quantity can be controlled.
2. Application container logs can generate log files first, and then soft-connect the application log files to the standard output, such as nginx; or logs can be directly printed to the standard output at startup, such as mysql.

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.