Docker cross-server communication overlay solution (Part 1) Consul single instance

Table of contents Scenario Task idea analyze Concept and selection A little test Environmental Description Precautions Hands-on test Citations Scenario The company's microservices are about to go online. Microservices are deployed using Docker containers. On the same host, all services are deployed. The IP and PORT registered to Nacos are the intranet IP and the port number defined in the Dockerfile. It seems that there is no problem. It can also be called through the gateway. Please note that there is a big premise: All service containers must be deployed on the same host! When the service instances are not deployed on the same host, for example, the gateway service is on server A and service a is on server B, and they are both registered with Nacos (or other registration centers), the IP addresses reported are all from the intranet. When an external request comes in, the gateway finds the corresponding intranet IP address of service a through the service list of Nacos, but the call fails. ps: How can the intranet be connected... Task Microservice containers can be on different servers and call each other idea Since the reported IP is from the intranet, I will just ask him to report the host machine's IP and port. Using Docker's host network mode Modify the deployment script to obtain the host IP and the set mapping port number when deploying the container through the shell Make Docker network interoperable analyze The following is an explanation of the problem according to the "ideas" section above. 1. After looking through the official documents and Github, there are two solutions: Fixed IP port, hard-coded the host IP and port in the configuration file: It seems to be solved, but the problem is that it cannot be horizontally expanded - barely usable Fixed the network card to prevent reporting the wrong IP port due to multiple network card environments: It is useless. After entering the container, `ifconfig` found that there are only two internal network cards, `eth0` and `lo` . The IP of the corresponding network card is the internal network IP. It is still useless. 2. Using Docker's Host network mode, you will find that the IP reported this time is indeed the host IP, but the port number is wrong... If you use the Java parameter to pass in the port number to be mapped through the shell, this is theoretically feasible. The only disadvantage is that `docker ps` can no longer directly see the port number, and you need to use `docker inspect` additionally - you can use 3. The mapped port number can be obtained, but the host's network card name is different and it is not flexible after being hard-coded. What if some are `eth0` and some are `ens33` ? There are many more unpredictable situations! ——Maybe it can be used 4. Share through some mature Docker container networks, but there will be some performance loss - completely available Concept and selection The most reliable way is to use Docker network sharing. With the help of search engines, I decided to use Overlay to achieve the effect. Here is a brief description of Overlay: When containers communicate across two hosts, they use the overlay network mode for communication; if the host is used, cross-host communication can also be achieved by directly using the physical IP address. Overlay will create a virtual network, such as the IP address 10.0.2.3. In this overlay network mode, there is an address similar to a service gateway, which forwards the packet to the address of the physical server, and finally reaches the IP address of another server through routing and switching. To implement an Overlay network, you need to introduce a KV database to save network status information, including Network, Endpoint, IP, etc. Consul, Etcd and ZooKeeper are all KV databases supported by Docker We use Consul here. Compared with other KV databases, the interface provided by Consul is convenient for management, so Consul is used here to implement Overlay By having each server's Docker daemon register its own IP address with Consul, you can share the Docker intranet. The intranet shared here is in Overlay network mode. Only containers using the same overlay network in the registered Docker environment can communicate with each other. ps: After creation, cross-server containers that do not use the overlay network cannot be pinged A little test Single-node Consul implements Overlay network using Docker image Environmental Description

Scenario
Task
idea
analyze
Concept and selection
A little test
Environmental Description
Precautions
Hands-on
test
Citations

Scenario

The company's microservices are about to go online. Microservices are deployed using Docker containers. On the same host, all services are deployed. The IP and PORT registered to Nacos are the intranet IP and the port number defined in the Dockerfile. It seems that there is no problem. It can also be called through the gateway. Please note that there is a big premise:

All service containers must be deployed on the same host!

When the service instances are not deployed on the same host, for example, the gateway service is on server A and service a is on server B, and they are both registered with Nacos (or other registration centers), the IP addresses reported are all from the intranet. When an external request comes in, the gateway finds the corresponding intranet IP address of service a through the service list of Nacos, but the call fails.

ps: How can the intranet be connected...

Task

Microservice containers can be on different servers and call each other

idea

Since the reported IP is from the intranet, I will just ask him to report the host machine's IP and port.
Using Docker's host network mode
Modify the deployment script to obtain the host IP and the set mapping port number when deploying the container through the shell
Make Docker network interoperable

analyze

The following is an explanation of the problem according to the "ideas" section above.

1. After looking through the official documents and Github, there are two solutions:

Fixed IP port, hard-coded the host IP and port in the configuration file: It seems to be solved, but the problem is that it cannot be horizontally expanded - barely usable
Fixed the network card to prevent reporting the wrong IP port due to multiple network card environments: It is useless. After entering the container, ifconfig found that there are only two internal network cards, eth0 and lo . The IP of the corresponding network card is the internal network IP. It is still useless.

2. Using Docker's Host network mode, you will find that the IP reported this time is indeed the host IP, but the port number is wrong... If you use the Java parameter to pass in the port number to be mapped through the shell, this is theoretically feasible. The only disadvantage is that docker ps can no longer directly see the port number, and you need to use docker inspect additionally - you can use

3. The mapped port number can be obtained, but the host's network card name is different and it is not flexible after being hard-coded. What if some are eth0 and some are ens33 ? There are many more unpredictable situations! ——Maybe it can be used

4. Share through some mature Docker container networks, but there will be some performance loss - completely available

Concept and selection

The most reliable way is to use Docker network sharing. With the help of search engines, I decided to use Overlay to achieve the effect.

Here is a brief description of Overlay:

When containers communicate across two hosts, they use the overlay network mode for communication; if the host is used, cross-host communication can also be achieved by directly using the physical IP address. Overlay will create a virtual network, such as the IP address 10.0.2.3. In this overlay network mode, there is an address similar to a service gateway, which forwards the packet to the address of the physical server, and finally reaches the IP address of another server through routing and switching.

To implement an Overlay network, you need to introduce a KV database to save network status information, including Network, Endpoint, IP, etc. Consul, Etcd and ZooKeeper are all KV databases supported by Docker

We use Consul here. Compared with other KV databases, the interface provided by Consul is convenient for management, so Consul is used here to implement Overlay

By having each server's Docker daemon register its own IP address with Consul, you can share the Docker intranet. The intranet shared here is in Overlay network mode. Only containers using the same overlay network in the registered Docker environment can communicate with each other.

ps: After creation, cross-server containers that do not use the overlay network cannot be pinged

A little test

Single-node Consul implements Overlay network using Docker image

Environmental Description

Server OS	Host IP	Docker version	Network card name
Ubuntu Server 18.04 LTS	192.168.87.133	18.09.6	ens33
Ubuntu Server 18.04 LTS	192.168.87.139	18.09.7	ens33

The Consul version to be used is 1.5.2, which has the smallest vulnerability currently on Docker Hub.

This test environment is suitable for Linux distributions managed by Systemd
Consul did not use the unofficial progrium/consul, mainly because this image is too old. If there were vulnerabilities four years ago, they could not be fixed in time, so I went to <hub.docker.com> to explore all the official pitfalls! 💔

Precautions

Each host running Docker must have a different hostname. You can use

$ sudo hostnamectl set-hostname your-new-hostname

The same hostname will cause the host dockers with the same name to be unable to communicate with each other

Hands-on

Prepare Consul to start on the 133 server using the image, so you can first configure the startup parameters of the Docker daemon to point to the 133 server

Modify the docker.service of servers 133 and 139 respectively

$ ifconfig
#The interfering network card has been removed, and the network card name is ens33
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 192.168.87.133 netmask 255.255.255.0 broadcast 192.168.87.255
        inet6 fe80::20c:29ff:fe02:e00a prefixlen 64 scopeid 0x20<link>
        ether 00:0c:29:02:e0:0a txqueuelen 1000 (Ethernet)
        RX packets 156739 bytes 233182466 (233.1 MB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 45173 bytes 2809606 (2.8 MB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

$ vim /etc/docker/daemon.json

Save and exit.

cluster-store : The leader address of the configured Consul. The single unit can be written directly. Other software should pay attention to the protocol.
cluster-advertise : Specify the listening network card and port, and also specify the IP:PORT to receive subscription messages

Another way is to modify docker.service directly, as shown below:

$ cd /etc/systemd/system/multi-user.target.wants
$ sudo vim docker.service

Find the word ExecStart= and add the following code at the end of this line

--cluster-store=consul://192.168.87.133:8500 --cluster-advertise=ens33:2375

The effect is as follows:

The operation effect is consistent with the above method

Then execute the command to restart the docker service. The operation method of the other server is the same. Pay attention to the name of the network card.

$ sudo systemctl daemon-reload && sudo systemctl restart docker

Start the Consul service on the 133 server

docker run -d --network host -h consul --name=consul --restart=always -e CONSUL_BIND_INTERFACE=ens33 consul:1.5.2

The reason for using host mode is to prevent some ports from not being mapped, and the only way to let Consul recognize the external network card is host mode. Here is a non-host method

$ docker run -di -h consul -p 8500:8500 --name=consul consul:1.5.2

Create a Docker overlay shared network

$ docker network create -d overlay my_overlay

The difference between this and creating a normal network is that the overlay mode network is specified. -d can also be written as --driver

Visit the Consul page, such as mine is 192.168.87.133:8500

Our configuration is at Key/Value.

Click docker -> nodes

The two nodes above are the values registered by two docker daemons.

test

Create two new centos containers on two servers, using the overlay network we just created

133 Server

$ docker run -di --network my_overlay --name mycentos1 centos:7

139 servers

$ docker run -di --network my_overlay --name mycentos2 centos:7

--net is spelled as --network . The one starting with -- can be omitted =

View the IP of the mycentos1 container on the 133 server

$ docker inspect -f "{{ .NetworkSettings.Networks.my_overlay.IPAddress}}" mycentos1
10.0.1.2

View the IP of the mycentos2 container on the 139 server

$ docker inspect -f "{{ .NetworkSettings.Networks.my_overlay.IPAddress}}" mycentos2
10.0.1.3

Ping the intranet IP of mycentos2 of server 139 from server 133 respectively

Ping is the same as the other way around, but it doesn't allow us to access it from the outside, but through the container on the same overlay network. If you don't believe me, let's try it as follows

133 Server

$ docker exec -it mycentos1 bash
# ping 10.0.1.3

The access is successful, there is no packet loss, and the reverse is the same. Due to limited space, I will not test it.

This at least shows that the current services are indeed interconnected. In the next article, we will develop a clustering method for production.

Citations

Technical Talk | Docker overlay network implementation

Docker container overlay network

Docker cross-host network - overlay

Day 22: Introduction to Docker Network (Part 3)

Consul KV

Docker Hub Consul

Github docker-library/docs

Github docker-consul

This article is a learning and practice article by Hellxz. It is forbidden to crawl third-party crawler websites such as Bubukou and Mami.

This concludes this article about the Docker cross-server communication Overlay solution (Part 1) - Consul single instance. For more relevant Docker Overlay network content, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope everyone will support 123WORDPRESS.COM in the future!

You may also be interested in: