Solution to the problem that docker logs cannot be retrieved

When checking the service daily, when I went to look at the container logs from portainer, I found a red exclamation mark in the upper right corner: Unable to retrieve container logs.

Because this problem has never occurred before, I first went to the server and used the command docker logs -f containerID to check the logs. I found that the logs could not be moved and were still stuck at a certain time.

After thinking for a while, I thought that it should not be a problem with the service log printing. I searched on Google first, but found that there was no result that matched my problem. Because logs can sometimes be collected and displayed, but some logs cannot, there should be a problem with the log engine set up by Docker.

I originally wanted to install a set of EFK, but I felt that the amount of logs was not large enough, so I did not modify the docker log engine and still used the default journald.

[root@ad-official xiaoxiao]# docker info|grep Logging
 WARNING: You're not using the default seccomp profile
Logging Driver: journald

There is such a description in the official documentation of journald:

man journald.conf ... RateLimitInterval=, RateLimitBurst= Configures the rate limiting that is applied to all messages generated on the system. If, in the time interval defined by RateLimitInterval=, more messages than specified in RateLimitBurst= are logged by a service, all further messages within the interval are dropped until the interval is over. A message about the number of dropped messages is generated. This rate limiting is applied per-service, so that two services which log do not interfere with each other's limits. Defaults to 1000 messages in 30s. The time specification for RateLimitInterval= may be specified in the following units: "s", "min", "h", "ms", "us". To turn off any kind of rate limiting, set either value to 0. ...

It is written here that by default only 1,000 logs can be received within 30 seconds. You can understand it from here, because a while ago we released a service with a daily log file size of almost 3G in Docker, which affected the logs of other services. A large number of logs were discarded by journald, so we can modify the configuration and there will be no problem.

Open the /etc/systemd/journald.conf file and change RateLimitBurst from the default 1000 to 5000, adjusting it according to your current log output:

[root@ad-official log]# cat /etc/systemd/journald.conf
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
RateLimitBurst=5000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
ForwardToSyslog=no
#ForwardToKMsg=no
#ForwardToConsole=no
ForwardToWall=no
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K

By the way, set ForwardToSyslog and ForwardToWall to no, because the default is yes, which will cause us to clean up the journal log files while the Syslog log files are not cleared, which will slowly fill up the disk.

Then restart journald to resume normal use: systemctl restart systemd-journald.service

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.