Detailed explanation of the reasons and solutions for Docker failing to start normally

Detailed explanation of the reasons and solutions for Docker failing to start normally

1. Abnormal performance of Docker startup:

1. The status is repeatedly restaring, check it with the command

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
21c09be88c11 docker.xxxx.cn:5000/xxx-tes/xxx_tes:1.0.6 "/usr/local/tomcat..." 9 days ago Restarting (1) Less than a second ago xxx10

2. There are obvious problems with Docker logs:

$docker logs [container name/container ID]

2. Possible reasons for Docker startup abnormality:

2.1. Insufficient memory

Docker requires at least 2G of memory to start. First, execute the free -mh command to check whether the remaining memory is sufficient.

View memory directly

$free -mh
      total used free shared buff/cache available
Mem: 15G 14G 627M 195M 636M 726M
Swap: 0B 0B 0B

Analyze logs

Sometimes the memory is overloaded for a moment, causing some processes to be killed. It seems that the memory is sufficient, but in fact Docker will restart repeatedly. Further analysis is needed through the Docker log and system log information:

Analyzing Docker logs

Check the docker log to see the memory overflow information. You need to read it carefully to find the information. It is not at the bottom

$docker logs [container name/container ID]|less 
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# An error report file with more information is saved as:
# //hs_err_pid1.log
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000769990000, 1449590784, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# Can not save log file, dump to screen..
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1449590784 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2756), pid=1, tid=140325689620224
#
# JRE version: (7.0_79-b15) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: //core or core.1
#

Analyze system logs

Checking the system log, I found a large number of records of processes being killed due to memory overflow.

$grep -i 'Out of Memory' /var/log/messages
Apr 7 10:04:02 centos106 kernel: Out of memory: Kill process 1192 (java) score 54 or sacrifice child
Apr 7 10:08:00 centos106 kernel: Out of memory: Kill process 2301 (java) score 54 or sacrifice child
Apr 7 10:09:59 centos106 kernel: Out of memory: Kill process 28145 (java) score 52 or sacrifice child
Apr 7 10:20:40 centos106 kernel: Out of memory: Kill process 2976 (java) score 54 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3577 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3631 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3634 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3640 (java) score 47 or sacrifice child
Apr 7 10:21:08 centos106 kernel: Out of memory: Kill process 3654 (java) score 47 or sacrifice child
Apr 7 10:27:27 centos106 kernel: Out of memory: Kill process 6998 (java) score 51 or sacrifice child
Apr 7 10:27:28 centos106 kernel: Out of memory: Kill process 7027 (java) score 52 or sacrifice child
Apr 7 10:28:10 centos106 kernel: Out of memory: Kill process 7571 (java) score 42 or sacrifice child
Apr 7 10:28:10 centos106 kernel: Out of memory: Kill process 7586 (java) score 42 or sacrifice child

2.2. Port Conflict

The Docker listening port has been occupied by other processes. This problem is prone to occur in newly deployed services or when deploying new background services on the original machine. Therefore, before deployment, you should execute a command to check whether the port is occupied. If it is found to be occupied after going online, you should change it to an available port and restart it.

Check command: $netstat -nltp|grep [planned port number]

3. Countermeasures

3.1. Countermeasures for insufficient memory:

Countermeasure 1:
3.1.1 After running for too long, the saltstack minion may occupy a large amount of memory and need to be restarted. The reboot command may not work sometimes. Mainly check the running status. If it fails to stop, restart it.

Countermeasure 2:
3.2.2 If the ELK log collection program or other Java processes occupy too much space, use the top and ps commands to check, carefully determine the role of the process, and stop the relevant process without affecting the business;

Strategy 3:
Release the occupied memory (buff/cache):
$sync #Write memory data to disk
$echo 3 > /proc/sys/vm/drop_caches #Release the occupied memory

Countermeasure 4:
Sometimes it is not that the buff/cache is too high that causes insufficient memory, but that the memory is indeed consumed by many necessary processes. In this case, it is necessary to consider and solve the problem from the perspective of machine resource allocation and usage.

3.2 Countermeasures for port conflicts

Countermeasure 1:
Generally, this problem is prone to occur in newly deployed services, or when deploying new background services on the original machine. Therefore, before deployment, you should execute a command to check whether the port is occupied. If it is found to be occupied after going online, you should change it to an available port and restart it.
Check command: $netstat -nltp|grep [planned port number]

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.

You may also be interested in:
  • CentOS7 installs docker to solve the problem of not being able to start
  • Detailed explanation of how Docker starts a CentOS image
  • Solution and implementation steps for CentOS7 docker service failure to start
  • How to start multiple services simultaneously in Docker
  • How to enter the startup container in Docker
  • The difference between two ways to create a docker image and start the container (summary)
  • Solution to MySQL initialization and startup failure in Docker
  • Example of how to automatically start an application service in a Docker container
  • How to implement Docker container self-start
  • How to create, start, and stop a Docker container

<<:  Use of Vue filters and custom instructions

>>:  MySQL 8.0.11 MSI version installation and configuration graphic tutorial

Recommend

Several ways to solve the problem of floating causing the height of the parent element to collapse in CSS

1. Document flow and floating 1. What is document...

Installing Win10 system on VMware workstation 14 pro

This article introduces how to install the system...

Summary of some tips on MySQL index knowledge

Table of contents 1. Basic knowledge of indexing ...

How the Linux kernel breaks into the process address space and modifies the process memory

Isolation of process address spaces is a notable ...

Learn the basics of JavaScript DOM operations in one article

DOM Concepts DOM: document object model: The docu...

Discussion on default margin and padding values ​​of common elements

Today we discussed the issue of what the margin v...

Native JS to achieve book flipping effects

This article shares with you a book flipping effe...

Quick solution for forgetting MySQL8 password

Preface When we forget the MySQL database passwor...

Example code for making the pre tag automatically wrap

The pre element defines preformatted text. Text en...

border-radius method to add rounded borders to elements

border-radius:10px; /* All corners are rounded wi...

A brief discussion on four solutions for Vue single page SEO

Table of contents 1.Nuxt server-side rendering ap...

Implementation of Docker deployment of Tomcat and Web applications

1. Download docker online yum install -y epel-rel...

What are the benefits of semantic HTML structure?

one: 1. Semantic tags are just HTML, there is no ...

MySQL 8.0.16 installation and configuration graphic tutorial under macOS

This article shares the installation and configur...

Install CentOS 7 on VMware14 Graphic Tutorial

Introduction to CentOS CentOS is an enterprise-cl...