Detailed explanation of the reasons and solutions for Docker failing to start normally

Detailed explanation of the reasons and solutions for Docker failing to start normally

1. Abnormal performance of Docker startup:

1. The status is repeatedly restaring, check it with the command

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
21c09be88c11 docker.xxxx.cn:5000/xxx-tes/xxx_tes:1.0.6 "/usr/local/tomcat..." 9 days ago Restarting (1) Less than a second ago xxx10

2. There are obvious problems with Docker logs:

$docker logs [container name/container ID]

2. Possible reasons for Docker startup abnormality:

2.1. Insufficient memory

Docker requires at least 2G of memory to start. First, execute the free -mh command to check whether the remaining memory is sufficient.

View memory directly

$free -mh
      total used free shared buff/cache available
Mem: 15G 14G 627M 195M 636M 726M
Swap: 0B 0B 0B

Analyze logs

Sometimes the memory is overloaded for a moment, causing some processes to be killed. It seems that the memory is sufficient, but in fact Docker will restart repeatedly. Further analysis is needed through the Docker log and system log information:

Analyzing Docker logs

Check the docker log to see the memory overflow information. You need to read it carefully to find the information. It is not at the bottom

$docker logs [container name/container ID]|less 
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# An error report file with more information is saved as:
# //hs_err_pid1.log
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# Can not save log file, dump to screen..
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2756), pid=1, tid=140325689620224
#
# JRE version: (7.0_79-b15) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: //core or core.1
#

Analyze system logs

Checking the system log, I found a large number of records of processes being killed due to memory overflow.

$grep -i 'Out of Memory' /var/log/messages
Apr 7 10:04:02 centos106 kernel: Out of memory: Kill process 1192 (java) score 54 or sacrifice child
Apr 7 10:08:00 centos106 kernel: Out of memory: Kill process 2301 (java) score 54 or sacrifice child
Apr 7 10:09:59 centos106 kernel: Out of memory: Kill process 28145 (java) score 52 or sacrifice child
Apr 7 10:20:40 centos106 kernel: Out of memory: Kill process 2976 (java) score 54 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3577 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3631 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3634 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3640 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3654 (java) score 47 or sacrifice child
Apr 7 10:27:27 centos106 kernel: Out of memory: Kill process 6998 (java) score 51 or sacrifice child
Apr 7 10:27:28 centos106 kernel: Out of memory: Kill process 7027 (java) score 52 or sacrifice child
Apr 7 10:28:10 centos106 kernel: Out of memory: Kill process 7571 (java) score 42 or sacrifice child
Apr 7 10:28:10 centos106 kernel: Out of memory: Kill process 7586 (java) score 42 or sacrifice child

2.2. Port Conflict

The Docker listening port has been occupied by other processes. This problem is prone to occur in newly deployed services or when deploying new background services on the original machine. Therefore, before deployment, you should execute a command to check whether the port is occupied. If it is found to be occupied after going online, you should change it to an available port and restart it.

Check command: $netstat -nltp|grep [planned port number]

3. Countermeasures

3.1. Countermeasures for insufficient memory:

Countermeasure 1:
3.1.1 After running for too long, the saltstack minion may occupy a large amount of memory and need to be restarted. The reboot command may not work sometimes. Mainly check the running status. If it fails to stop, restart it.

Countermeasure 2:
3.2.2 If the ELK log collection program or other Java processes occupy too much space, use the top and ps commands to check, carefully determine the role of the process, and stop the relevant process without affecting the business;

Strategy 3:
Release the occupied memory (buff/cache):
$sync #Write memory data to disk
$echo 3 > /proc/sys/vm/drop_caches #Release the occupied memory

Countermeasure 4:
Sometimes it is not that the buff/cache is too high that causes insufficient memory, but that the memory is indeed consumed by many necessary processes. In this case, it is necessary to consider and solve the problem from the perspective of machine resource allocation and usage.

3.2 Countermeasures for port conflicts

Countermeasure 1:
Generally, this problem is prone to occur in newly deployed services, or when deploying new background services on the original machine. Therefore, before deployment, you should execute a command to check whether the port is occupied. If it is found to be occupied after going online, you should change it to an available port and restart it.
Check command: $netstat -nltp|grep [planned port number]

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.

You may also be interested in:
  • CentOS7 installs docker to solve the problem of not being able to start
  • Detailed explanation of how Docker starts a CentOS image
  • Solution and implementation steps for CentOS7 docker service failure to start
  • How to start multiple services simultaneously in Docker
  • How to enter the startup container in Docker
  • The difference between two ways to create a docker image and start the container (summary)
  • Solution to MySQL initialization and startup failure in Docker
  • Example of how to automatically start an application service in a Docker container
  • How to implement Docker container self-start
  • How to create, start, and stop a Docker container

<<:  Use of Vue filters and custom instructions

>>:  MySQL 8.0.11 MSI version installation and configuration graphic tutorial

Recommend

Detailed explanation of HTML tables

Function: data display, table application scenari...

Quickly solve the Chinese input method problem under Linux

Background: I'm working on asset reporting re...

Mini Program to Implement Simple List Function

This article example shares the specific code of ...

How to set mysql permissions using phpmyadmin

Table of contents Step 1: Log in as root user. St...

How to make the height of child div fill the remaining space of parent container in CSS

1. Use floating method Effect picture: The code i...

Let's talk about parameters in MySQL

Preface: In some previous articles, we often see ...

Linux remote login implementation tutorial analysis

Linux is generally used as a server, and the serv...

Example of how to implement local fuzzy search function in front-end JavaScript

Table of contents 1. Project Prospects 2. Knowled...

CSS3 to achieve simple white cloud floating background effect

This is a very simple pure CSS3 white cloud float...

Vue two same-level components to achieve value transfer

Vue components are connected, so it is inevitable...

How to deploy Go web applications using Docker

Table of contents Why do we need Docker? Docker d...

JavaScript function syntax explained

Table of contents 1. Ordinary functions 2. Arrow ...

A brief discussion on value transfer between Vue components (including Vuex)

Table of contents From father to son: Son to Fath...

What should I do if I can't view the source file of a web page?

Q: Whether using Outlook or IE, when you right-cl...

Nginx reverse proxy forwards port 80 requests to 8080

Let's first understand a wave of concepts, wh...