Detailed explanation of the reasons and solutions for Docker failing to start normally

Detailed explanation of the reasons and solutions for Docker failing to start normally

1. Abnormal performance of Docker startup:

1. The status is repeatedly restaring, check it with the command

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
21c09be88c11 docker.xxxx.cn:5000/xxx-tes/xxx_tes:1.0.6 "/usr/local/tomcat..." 9 days ago Restarting (1) Less than a second ago xxx10

2. There are obvious problems with Docker logs:

$docker logs [container name/container ID]

2. Possible reasons for Docker startup abnormality:

2.1. Insufficient memory

Docker requires at least 2G of memory to start. First, execute the free -mh command to check whether the remaining memory is sufficient.

View memory directly

$free -mh
      total used free shared buff/cache available
Mem: 15G 14G 627M 195M 636M 726M
Swap: 0B 0B 0B

Analyze logs

Sometimes the memory is overloaded for a moment, causing some processes to be killed. It seems that the memory is sufficient, but in fact Docker will restart repeatedly. Further analysis is needed through the Docker log and system log information:

Analyzing Docker logs

Check the docker log to see the memory overflow information. You need to read it carefully to find the information. It is not at the bottom

$docker logs [container name/container ID]|less 
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# An error report file with more information is saved as:
# //hs_err_pid1.log
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# Can not save log file, dump to screen..
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2756), pid=1, tid=140325689620224
#
# JRE version: (7.0_79-b15) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: //core or core.1
#

Analyze system logs

Checking the system log, I found a large number of records of processes being killed due to memory overflow.

$grep -i 'Out of Memory' /var/log/messages
Apr 7 10:04:02 centos106 kernel: Out of memory: Kill process 1192 (java) score 54 or sacrifice child
Apr 7 10:08:00 centos106 kernel: Out of memory: Kill process 2301 (java) score 54 or sacrifice child
Apr 7 10:09:59 centos106 kernel: Out of memory: Kill process 28145 (java) score 52 or sacrifice child
Apr 7 10:20:40 centos106 kernel: Out of memory: Kill process 2976 (java) score 54 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3577 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3631 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3634 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3640 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3654 (java) score 47 or sacrifice child
Apr 7 10:27:27 centos106 kernel: Out of memory: Kill process 6998 (java) score 51 or sacrifice child
Apr 7 10:27:28 centos106 kernel: Out of memory: Kill process 7027 (java) score 52 or sacrifice child
Apr 7 10:28:10 centos106 kernel: Out of memory: Kill process 7571 (java) score 42 or sacrifice child
Apr 7 10:28:10 centos106 kernel: Out of memory: Kill process 7586 (java) score 42 or sacrifice child

2.2. Port Conflict

The Docker listening port has been occupied by other processes. This problem is prone to occur in newly deployed services or when deploying new background services on the original machine. Therefore, before deployment, you should execute a command to check whether the port is occupied. If it is found to be occupied after going online, you should change it to an available port and restart it.

Check command: $netstat -nltp|grep [planned port number]

3. Countermeasures

3.1. Countermeasures for insufficient memory:

Countermeasure 1:
3.1.1 After running for too long, the saltstack minion may occupy a large amount of memory and need to be restarted. The reboot command may not work sometimes. Mainly check the running status. If it fails to stop, restart it.

Countermeasure 2:
3.2.2 If the ELK log collection program or other Java processes occupy too much space, use the top and ps commands to check, carefully determine the role of the process, and stop the relevant process without affecting the business;

Strategy 3:
Release the occupied memory (buff/cache):
$sync #Write memory data to disk
$echo 3 > /proc/sys/vm/drop_caches #Release the occupied memory

Countermeasure 4:
Sometimes it is not that the buff/cache is too high that causes insufficient memory, but that the memory is indeed consumed by many necessary processes. In this case, it is necessary to consider and solve the problem from the perspective of machine resource allocation and usage.

3.2 Countermeasures for port conflicts

Countermeasure 1:
Generally, this problem is prone to occur in newly deployed services, or when deploying new background services on the original machine. Therefore, before deployment, you should execute a command to check whether the port is occupied. If it is found to be occupied after going online, you should change it to an available port and restart it.
Check command: $netstat -nltp|grep [planned port number]

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.

You may also be interested in:
  • CentOS7 installs docker to solve the problem of not being able to start
  • Detailed explanation of how Docker starts a CentOS image
  • Solution and implementation steps for CentOS7 docker service failure to start
  • How to start multiple services simultaneously in Docker
  • How to enter the startup container in Docker
  • The difference between two ways to create a docker image and start the container (summary)
  • Solution to MySQL initialization and startup failure in Docker
  • Example of how to automatically start an application service in a Docker container
  • How to implement Docker container self-start
  • How to create, start, and stop a Docker container

<<:  Use of Vue filters and custom instructions

>>:  MySQL 8.0.11 MSI version installation and configuration graphic tutorial

Recommend

Sample code for partitioning and formatting a disk larger than 20TB on centos6

1. Server environment configuration: 1. Check dis...

Perfect solution to the problem of webpack packaging css background image path

Inside the style tag of the vue component, there ...

Draw a heart with CSS3

Achieve resultsRequirements/Functionality: How to...

Implementing license plate input function in WeChat applet

Table of contents Preface background Big guess Fi...

Complete steps to build a squid proxy server in linux

Preface This article mainly introduces the releva...

express project file directory description and detailed function description

app.js: startup file, or entry file package.json:...

How to implement a multi-terminal bridging platform based on websocket in JS

Table of contents 1. What to debug 2. Features of...

Vue implementation counter case

This article example shares the specific code of ...

How to skip errors in mysql master-slave replication

1. Traditional binlog master-slave replication, s...

Nginx routing forwarding and reverse proxy location configuration implementation

Three ways to configure Nginx The first method di...

Three ways to prevent MySQL from inserting duplicate data

Create a new table CREATE TABLE `person` ( `id` i...

Detailed explanation of the use of Vue card-style click-to-switch image component

This article shares the vue card-style click-to-s...

Writing and understanding of arrow functions and this in JS

Table of contents Preface 1. How to write functio...

Detailed explanation of Linux host name modification command

Linux change hostname command 1. If you only need...

How to set up URL link in Nginx server

For websites with an architecture like LNMP, they...