Detailed explanation of Docker container cross-host multi-network segment communication solution

1. MacVlan

There are many solutions to achieve cross-host network communication of Docker, such as the deployment of Consul service to achieve cross-host communication of Docker containers in the previous blog post.

Macvlan working principle:

Macvlan is a network interface supported by the Linux kernel. Required Linux builds are v3.9–3.19 and 4.0+;
By creating a Macvlan sub-interface for a physical network card, a physical network card is allowed to have multiple independent MAC addresses and IP addresses. The virtualized sub-interface will be directly exposed to the adjacent physical network. From the outside, it looks like the network cable is divided into multiple strands and connected to different hosts respectively;
After the physical network card receives the packet, it will determine whether the packet needs to be handed over to the virtual network card based on the destination MAC address of the received packet.

When the container needs to be directly connected to the physical network, Macvlan can be used. Macvlan itself does not create a network. In essence, it first makes the host physical network card work in 'promiscuous mode', so that the MAC address of the physical network card will be invalid, and the physical network card can receive all traffic in the layer 2 network. The next step is to create a virtual network card on this physical network card and assign a MAC address to the virtual network card to achieve multiple uses of one card. From the perspective of the physical network, each virtual network card is a separate interface.

Note when using Macvlan:

The container is directly connected to the physical network, which is responsible for allocating IP addresses. This may result in the exhaustion of physical network IP addresses. Another consequence is network performance issues. As more hosts are connected to the physical network, the proportion of broadcast packets increases rapidly, causing network performance to degrade.
A network on the host needs to work in 'promiscuous mode';
As mentioned above, the MAC address of a physical network card working in promiscuous mode will become invalid. Therefore, containers running in this mode cannot communicate with the external network, but it will not affect the communication between the host and the external network.
In the long run, bridge networks and overlay networks are better choices because virtual networks should be isolated from physical networks rather than shared.

Working diagram:

2. Configuration Example

Example 1 (implementing single-segment cross-host communication in containers based on macvlan)

Result:

Two centos 7.3 machines, each running a docker service;
The two Docker servers create the same MacVlan network, so that the containers on the Docker servers can communicate across hosts.

Start configuration

1. The first docker server is configured as follows

[root@docker01 ~]# ip link set ens33 promisc on # Enable promiscuous mode for ens33 network card [root@docker01 ~]# ip link show ens33 # Make sure the viewed information contains the following words in red2: ens33: <BROADCAST,MULTICAST,'PROMISC',UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
  link/ether 00:0c:29:9f:33:9f brd ff:ff:ff:ff:ff:ff
[root@docker01 ~]# docker network create -d macvlan --subnet 172.22.16.0/24 --gateway 172.22.16.1 -o pa
rent=ens33 mac_net1
#Create a macvlan network, specify the gateway, network segment and other information, "-o" specifies which network card to bind to [root@docker01 ~]# docker run -itd --name test1 --ip 172.22.16.10 --network mac_net1 busybox # Run a container based on the newly created macvlan network and specify its IP

Confirm the IP address of the running container

[root@docker01 ~]# docker exec test1 ip a # Check the IP and make sure the following red ones are the same as configured: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  inet 127.0.0.1/8 scope host lo
    valid_lft forever preferred_lft forever
6: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
  link/ether 02:42:ac:16:10:0a brd ff:ff:ff:ff:ff:ff
  inet '172.22.16.10/24' brd 172.22.16.255 scope global eth0
    valid_lft forever preferred_lft forever

2. The second docker server is configured as follows (basically similar to the first docker server)

[root@docker02 ~]# ip link set ens33 promisc on # Enable promiscuous mode [root@docker02 ~]# ip link show ens33 
2: ens33: <BROADCAST,MULTICAST,'PROMISC',UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
  link/ether 00:0c:29:b5:bc:ed brd ff:ff:ff:ff:ff:ff
[root@docker02 ~]# docker network create -d macvlan --subnet 172.22.16.0/24 --gateway=172.22.16.1 -o parent=ens33 mac_net1
#Create a macvlan with the same network segment and gateway as the first docker server. And bind it to the physical network card.
#In order to visually see that the macvlan on other docker servers is in the same network segment as this one. Therefore, it is recommended to set the network name to be the same.
[root@docker02 ~]# docker run -itd --name test2 --ip 172.22.16.11 --network mac_net1 busybox
#Run a container and specify it is based on the macvlan network #Note that its IP address should not conflict with the container IP address on other docker servers

Confirm the IP address of the running container

[root@docker02 ~]# docker exec test2 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  inet 127.0.0.1/8 scope host lo
    valid_lft forever preferred_lft forever
6: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
  link/ether 02:42:ac:16:10:0b brd ff:ff:ff:ff:ff:ff
  inet '172.22.16.11/24' brd 172.22.16.255 scope global eth0
    valid_lft forever preferred_lft forever

Use container test2 on the second docker server to ping container test1 on the first docker server

OK, cross-host container communication is achieved through macvlan. Since using promiscuous mode will cause the MAC address of the physical network card to become invalid, the container cannot communicate with the external network through this mode.

Example 2 (Solution for multi-segment cross-host network based on macvlan)

The achieved effects are as follows:

Two centos 7.3 machines, each running a docker service;
Each host creates two MacVlan segments for containers to use (172.10.16.0/24 and 172.20.16.0/24);
The first docker server runs containers test1 and test2, and the second docker server runs containers test3 and test4.
Finally, containers in the same network segment across hosts can communicate with each other.

Start configuration:

1. The first docker server is configured as follows

[root@docker01 ~]# ip link set ens33 promisc on # Enable promiscuous mode of ens33 network card #That is, enable multiple virtual interfaces of the network card 
[root@docker01 ~]# ip link show ens33 # Make sure the information you view contains the following words in red2: ens33: <BROADCAST,MULTICAST,'PROMISC',UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
  link/ether 00:0c:29:9f:33:9f brd ff:ff:ff:ff:ff:ff
[root@docker01 ~]# modinfo 8021q
 # Check whether the 8021q module is loaded. If there is information returned, it means that the module has been loaded

The information returned by the modinfo 8021q command is as follows

[root@docker01 ~]# modprobe 8021q #If the 8021q module is not loaded, execute this command [root@docker01 ~]# cd /etc/sysconfig/network-scripts/
[root@docker01 network-scripts]# vim ifcfg-ens33 
           ................
BOOTPROTO=manual # Change this configuration item to "manual", which also means manual [root@docker01 network-scripts]# cp -p ifcfg-ens33 ifcfg-ens33.10 # Copy a network card configuration file, -p retains the properties of the original file [root@docker01 network-scripts]# vim ifcfg-ens33.10 
BOOTPROTO=none 
NAME=ens33.10 #Note the name change DEVICE=ens33.10 #Note the name change ONBOOT=yes
IPADDR=192.168.10.11 # Set an IP for the virtual network card
PREFIX=24
GATEWAY=192.168.10.2
VLAN=yes
[root@docker01 network-scripts]# cp ifcfg-ens33.10 ifcfg-ens33.20 
[root@docker01 network-scripts]# vim ifcfg-ens33.20 
BOOTPROTO=none
NAME=ens33.20
DEVICE=ens33.20
ONBOOT=yes
IPADDR=192.168.20.10 # Note that the IP here is not in the same network segment as ens33.10 PREFIX=24
GATEWAY=192.168.20.2
VLAN=yes
[root@docker01 network-scripts]# ifdown ens33;ifup ens33 #Restart the network card to make the changes take effect[root@docker01 network-scripts]# ifup ens33.10 # Start the network card[root@docker01 network-scripts]# ifup ens33.20 # Start[root@docker01 ~]# docker network create -d macvlan --subnet 172.10.16.0/24 --gateway 172.10.16.1 -o parent=ens33.10 mac_net10
#Create a macvlan network, define a network segment, gateway and bind it to ens33.10
[root@docker01 ~]# docker network create -d macvlan --subnet 172.20.16.0/24 --gateway 172.20.16.1 -o parent=ens33.20 mac_net20
#Create a macvlan network, define a network segment, gateway and bind it to ens33.20
#Next, run a container based on the macvlan network just created

2. The second docker server is configured as follows (basically similar to the first operation, pay attention to the IP not to conflict)

[root@docker02 ~]# ip link set ens33 promisc on # Enable promiscuous mode [root@docker02 ~]# ip link show ens33 
2: ens33: <BROADCAST,MULTICAST,'PROMISC',UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
  link/ether 00:0c:29:b5:bc:ed brd ff:ff:ff:ff:ff:ff
[root@docker02 ~]# modinfo 8021q
Please refer to Figure 1 for the returned information

[root@docker02 ~]# modprobe 8021q #If the 8021q module is not loaded, execute this command [root@docker02 ~]# cd /etc/sysconfig/network-scripts/
[root@docker02 network-scripts]# vim ifcfg-ens33 
      ...............#Omit some content BOOTPROTO=manual    
[root@docker02 network-scripts]# scp [email protected]:/etc/sysconfig/network-scripts/ifcfg-ens33.* . # Pay attention to the trailing “.”
ifcfg-ens33.10 100% 128 83.4KB/s 00:00  
ifcfg-ens33.20 100% 124 75.0KB/s 00:00 
[root@docker02 network-scripts]# vim ifcfg-ens33.10 
BOOTPROTO=none
NAME=ens33.10
DEVICE=ens33.10
ONBOOT=yes
IPADDR=192.168.10.11 # Change the IP to avoid conflict with the first one PREFIX=24
GATEWAY=192.168.10.2
VLAN=yes
[root@docker02 network-scripts]# vim ifcfg-ens33.20 

BOOTPROTO=none
NAME=ens33.20
DEVICE=ens33.20
ONBOOT=yes
IPADDR=192.168.20.11
PREFIX=24
GATEWAY=192.168.20.2
VLAN=yes
[root@docker02 network-scripts]# ifdown ens33;ifup ens33 # Restart the network card to make the configuration effective [root@docker02 network-scripts]# ifup ens33.10 # Start the network card [root@docker02 network-scripts]# ifup ens33.20
#Next, create a macvlan network, the same as the network created by the first docker server [root@docker02 ~]# docker network create -d macvlan --subnet 172.10.16.0/24 --gateway 172.10.16.1 -o parent=ens33.10 mac_net10
[root@docker02 ~]# docker network create -d macvlan --subnet 172.20.16.0/24 --gateway 172.20.16.1 -o parent=ens33.20 mac_net20 
[root@docker02 ~]# docker run -itd --name test3 --network mac_net10 --ip 172.10.16.11 busybox
[root@docker02 ~]# docker run -itd --name test4 --network mac_net20 --ip 172.20.16.21 busybox

After this configuration, you can perform a ping test. If the configuration is correct, test3 should be able to communicate with test1 (because they are both based on the mac_net10 network); test4 should be able to communicate with test2 (similarly).

But test3 and test1 cannot communicate with test4 and test2 (because they are not based on the same virtual LAN).

Container test3 pings container test1 test (Note: If you use a vmware virtual machine for testing, due to the characteristics of vmware, you need to change its network adapter to "bridge mode" instead of NAT mode, etc. Otherwise, communication will not be possible)

Container test4 pings container test2 test:

At this point, cross-host network multi-segment has been implemented. Similarly, each container cannot communicate with the external network. If you have patience, it is recommended to read the official Docker documentation

This is the end of this article about the Docker container cross-host multi-segment communication solution. For more relevant Docker container cross-host multi-segment communication content, please search 123WORDPRESS.COM's previous articles or continue to browse the following related articles. I hope everyone will support 123WORDPRESS.COM in the future!

You may also be interested in:

Operations of communication between Docker containers and external network communication
Implementation of Docker container connection and communication
Detailed explanation of direct routing in cross-host communication of Docker containers
Detailed explanation of how Docker containers communicate across hosts
Detailed explanation of a method of communication between docker containers
How to implement communication between Docker containers

<<: Solution to MySQL root password error number 1045

>>: Encapsulate the navigation bar component with Vue

Detailed explanation of Docker container cross-host multi-network segment communication solution

Solution to win10 without Hyper-V

VM VirtualBox virtual machine mount shared folder

How to modify the "Browse" button of the html form to upload files

Summary of basic usage of $ symbol in Linux

How to add automatic completion commands for docker and kubectl on Mac

A brief discussion on MySQL count of rows

CentOS7 installation zabbix 4.0 tutorial (illustration and text)

Solve MySQL startup error: ERROR 2003 (HY000): Can't connect to MySQL server on 'localhost' (10061)

Front-end implementation of GBK and GB2312 encoding and decoding of strings (summary)

N ways to cleverly implement adaptive dividers with CSS

Recommend

11 Reasons Why Bootstrap Is So Popular

Detailed explanation of JavaScript operation mechanism and a brief discussion on Event Loop

Bootstrap 3.0 study notes grid system case

How to import Tomcat source code into idea

Detailed analysis of the MySQL slow log opening method and storage format

Linux kernel device driver Linux kernel basic notes summary

Docker enables multiple port mapping commands

How to select all child elements and add styles to them in CSS

A brief analysis of kubernetes controllers and labels

Semantics, writing, and best practices of link A

Notes on the MySQL database backup process

Mysql get table comment field operation

Vue realizes the function of book shopping cart

Analysis of Context application scenarios in React

Teach you a trick to permanently solve the problem of MySQL inserting Chinese characters