Implementation of Docker to build Zookeeper&Kafka cluster

I've been learning Kafka recently. When I was preparing to test the cluster status, I felt that it was too troublesome to open three virtual machines or three different port numbers in one virtual machine (well... mainly because I was lazy).

Environment Preparation

A computer with internet access and a CentOS7 virtual machine

Why Use Virtual Machines? Because I use a laptop, the IP changes every time I connect to the network, and I have to modify the configuration file all the time, which is too cumbersome and inconvenient for testing. (This problem can be avoided by using Docker virtual network, but I didn’t know it during the experiment)

Docker Installation

If you have already installed Docker, please ignore this step.

Docker supports the following CentOS versions:
CentOS 7 (64-bit): The system must be 64-bit and the kernel version must be 3.10 or later.
CentOS 6.5 (64-bit) or later: The system must be 64-bit and the kernel version must be 2.6.32-431 or later.
CentOS only supports Docker in the kernel included in the distribution.

yum installation

Docker requires the CentOS kernel version to be higher than 3.10. Check the prerequisites above to verify whether your CentOS version supports Docker.

# Check the kernel version $ uname -a

#Install Docker
$ yum -y install docker

#Start the Docker background service$ service docker start

# Since there is no hello-world image locally, a hello-world image will be downloaded and run in the container.
$ docker run hello-world

Script installation

Make sure the yum package is updated to the latest version.

$ sudo yum update

Get and execute the Docker installation script.

$ curl -fsSL https://get.docker.com -o get-docker.sh
# Executing this script will add the docker.repo repository and install Docker.
$ sudo sh get-docker.sh

Start Docker

$ sudo systemctl start docker

# Verify that Docker is installed successfully and execute a test image in the container.
$ sudo docker run hello-world
$ docker ps

Mirror Acceleration

When I was first asked to configure a domestic mirror source, I refused. However, after using it, I found that the download speed increased duang~ . Therefore, it is strongly recommended that you configure a domestic mirror source.

Open/create the /etc/docker/daemon.json file and add the following content:

{
 "registry-mirrors": ["http://hub-mirror.c.163.com"]
}

Zookeeper cluster construction

Zookeeper image: zookeeper:3.4

Image preparation

$ docker pull zookeeper:3.4

To find images, go to https://hub.docker.com/

docker pull images:TAG // represents pulling the image image of TAG version

Create a separate Zookeeper container

We first create a separate Zookeeper node in the simplest way, and then we create other nodes based on this example.

$ docker run --name zookeeper -p 2181:2181 -d zookeeper:3.4

By default, the configuration file in the container is /conf/zoo.cfg , and the data and log directories are /data and /datalog by default. If necessary, the above directories can be mapped to the host.

Parameter Explanation

--name: Specify the container name
-p: Assign port numbers to ports exposed by the container
-d: Run the container in the background and print the container ID

Cluster construction

The creation method of Zookeeper containers for other nodes is similar to that of independent containers. It should be noted that the node id and the configuration of multiple nodes in the modification file should be specified separately. The corresponding creation commands are as follows:

Create a new docker network

$ docker network create zoo_kafka
$ docker network ls

Zookeeper Container 1

$ docker run -d \
   --restart=always \
   -v /opt/docker/zookeeper/zoo1/data:/data \
   -v /opt/docker/zookeeper/zoo1/datalog:/datalog \
   -e ZOO_MY_ID=1 \
   -p 2181:2181 \
   -e ZOO_SERVERS="server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888" \
   --name=zoo1 \
   --net=viemall-zookeeper \
   --privileged \
   zookeeper:3.4

Zookeeper Container 2

$ docker run -d \
   --restart=always \
   -v /opt/docker/zookeeper/zoo2/data:/data \
   -v /opt/docker/zookeeper/zoo2/datalog:/datalog \
   -e ZOO_MY_ID=2 \
   -p 2182:2181 \
   -e ZOO_SERVERS="server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888" \
   --name=zoo2 \
   --net=viemall-zookeeper \
   --privileged \
   zookeeper:3.4

Zookeeper Container 3

$ docker run -d \
   --restart=always \
   -v /opt/docker/zookeeper/zoo3/data:/data \
   -v /opt/docker/zookeeper/zoo3/datalog:/datalog \
   -e ZOO_MY_ID=3 \
   -p 2183:2181 \
   -e ZOO_SERVERS="server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888" \
   --name=zoo3 \
   --net=viemall-zookeeper \
   --privileged \
   zookeeper:3.4

Although this method also achieves what we want, the steps are too cumbersome and difficult to maintain (late stage of laziness), so we use docker-compose to achieve it.

Docker-compose builds a zookeeper cluster

Create a new docker network

$ docker network create viemall-zookeeper
$ docker network ls

Write the docker-compose.yml script

Directions:

Install docker-compose

# Get the script$ curl -L https://github.com/docker/compose/releases/download/1.25.0-rc2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
# Grant execution permissions $chmod +x /usr/local/bin/docker-compose

Create a new docker-compose.yml file in any directory and copy the following content

Execute the command docker-compose up -d

docker-compose.yml download address: https://github.com/JacianLiu/docker-compose/tree/master/zookeeper
docker-compose.yml details

version: '2'
services:
 zoo1:
  image: zookeeper:3.4 # Image name restart: always # Automatically restart when an error occurs hostname: zoo1
  container_name: zoo1
  privileged: true
  ports: #Port - 2181:2181
  volumes: #Mount data volume - ./zoo1/data:/data
   - ./zoo1/datalog:/datalog 
  environment:
   TZ: Asia/Shanghai
   ZOO_MY_ID: 1 # Node ID
   ZOO_PORT: 2181 # zookeeper port number ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 # zookeeper node list networks:
   default:
    ipv4_address: 172.23.0.11

 zoo2:
  image: zookeeper:3.4
  restart: always
  hostname: zoo2
  container_name: zoo2
  privileged: true
  ports:
   - 2182:2181
  volumes:
   - ./zoo2/data:/data
   - ./zoo2/datalog:/datalog
  environment:
   TZ: Asia/Shanghai
   ZOO_MY_ID: 2
   ZOO_PORT: 2181
   ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
  networks:
   default:
    ipv4_address: 172.23.0.12

 zoo3:
  image: zookeeper:3.4
  restart: always
  hostname: zoo3
  container_name: zoo3
  privileged: true
  ports:
   - 2183:2181
  volumes:
   - ./zoo3/data:/data
   - ./zoo3/datalog:/datalog
  environment:
   TZ: Asia/Shanghai
   ZOO_MY_ID: 3
   ZOO_PORT: 2181
   ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
  networks:
   default:
    ipv4_address: 172.23.0.13

networks:
 default:
  external:
   name: zoo_kafka

verify

From the figure we can see that there is a Leader and two Flower . So far our Zookeeper cluster has been built.

Kafka cluster construction

With the above foundation, is it still a problem to build a Kafka cluster? In fact, it’s just that the values of several variables are different.

With the above example, you don’t need to bother with a single-node Kafka . You can directly use the docker-compose method to deploy three nodes. In fact, the methods are similar. As mentioned above, they are just some different properties. At this time, we don’t need to create a new Docker network. You can directly use the network created when building the Zookeeper cluster!

Environment Preparation

Kafka image: wurstmeister/kafka
Kafka-Manager image: sheepkiller/kafka-manager

# If no version is specified, the latest version of the image will be pulled by default. docker pull wurstmeister/kafka
docker pull sheepkiller/kafka-manager

Write the docker-compose.yml script

Directions:

Install docker-compose

# Get the script$ curl -L https://github.com/docker/compose/releases/download/1.25.0-rc2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
# Grant execution permissions $chmod +x /usr/local/bin/docker-compose

Create a new docker-compose.yml file in any directory and copy the following content

Execute the command docker-compose up -d

docker-compose.yml download address: https://github.com/JacianLiu/docker-compose/tree/master/zookeeper
docker-compose.yml details

version: '2'

services:
 broker1:
  image: wurstmeister/kafka
  restart: always
  hostname: broker1
  container_name: broker1
  privileged: true
  ports:
   - "9091:9092"
  environment:
   KAFKA_BROKER_ID: 1
   KAFKA_LISTENERS: PLAINTEXT://broker1:9092
   KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker1:9092
   KAFKA_ADVERTISED_HOST_NAME: broker1
   KAFKA_ADVERTISED_PORT: 9092
   KAFKA_ZOOKEEPER_CONNECT: zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1
   JMX_PORT: 9988
  volumes:
   - /var/run/docker.sock:/var/run/docker.sock
   - ./broker1:/kafka/kafka\-logs\-broker1
  external_links:
  - zoo1
  - zoo2
  - zoo3
  networks:
   default:
    ipv4_address: 172.23.0.14

 broker2:
  image: wurstmeister/kafka
  restart: always
  hostname: broker2
  container_name: broker2
  privileged: true
  ports:
   - "9092:9092"
  environment:
   KAFKA_BROKER_ID: 2
   KAFKA_LISTENERS: PLAINTEXT://broker2:9092
   KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker2:9092
   KAFKA_ADVERTISED_HOST_NAME: broker2
   KAFKA_ADVERTISED_PORT: 9092
   KAFKA_ZOOKEEPER_CONNECT: zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1
   JMX_PORT: 9988
  volumes:
   - /var/run/docker.sock:/var/run/docker.sock
   - ./broker2:/kafka/kafka\-logs\-broker2
  external_links: #Connect to containers outside this compose file
  - zoo1
  - zoo2
  - zoo3
  networks:
   default:
    ipv4_address: 172.23.0.15

 broker3:
  image: wurstmeister/kafka
  restart: always
  hostname: broker3
  container_name: broker3
  privileged: true
  ports:
   - "9093:9092"
  environment:
   KAFKA_BROKER_ID: 3
   KAFKA_LISTENERS: PLAINTEXT://broker3:9092
   KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker3:9092
   KAFKA_ADVERTISED_HOST_NAME: broker3
   KAFKA_ADVERTISED_PORT: 9092
   KAFKA_ZOOKEEPER_CONNECT: zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1
   JMX_PORT: 9988
  volumes:
   - /var/run/docker.sock:/var/run/docker.sock
   - ./broker3:/kafka/kafka\-logs\-broker3
  external_links: #Connect to containers outside this compose file
  - zoo1
  - zoo2
  - zoo3
  networks:
   default:
    ipv4_address: 172.23.0.16

 kafka-manager:
  image: sheepkiller/kafka-manager:latest
  restart: always
  container_name: kafka-manager
  hostname: kafka-manager
  ports:
   - "9000:9000"
  links: # Connect to the container created by this compose file
   - broker1
   - broker2
   - broker3
  external_links: #Connect to containers outside this compose file
   - zoo1
   - zoo2
   - zoo3
  environment:
   ZK_HOSTS: zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1
   KAFKA_BROKERS: broker1:9092,broker2:9092,broker3:9092
   APPLICATION_SECRET: letmein
   KM_ARGS: -Djava.net.preferIPv4Stack=true
  networks:
   default:
    ipv4_address: 172.23.0.10

networks:
 default:
  external: # Use the created network name: zoo_kafka

verify

We open the management page of kafka-manager , the access path is, host ip:9000;

If shown, fill in the address of the Zookeeper cluster, scroll to the bottom and click save

Click on the cluster you just added, and you can see that there are three nodes in the cluster.

Problems encountered during the construction process

Mounting the data volume causes an infinite restart. Checking log prompts: chown: changing ownership of '/var/lib/mysql/....': Permission denied

Solution:

Add --privileged=true to docker run to give the container specific permissions
Temporarily disable selinux: setenforce 0
Add selinux rules to change the security text of the directory to be mounted

kafka-manager reports jmx related errors,

Solution:

Add the environment variable JMX_PORT=port to each kafka node
After adding it, I found that I couldn't connect, and it was a network connection problem. So I exposed each jmx port and then let the firewall pass, which solved the problem.
It is best to set KAFKA_ADVERTISED_HOST_NAME to the IP address of the host machine. Code or tools outside the host machine can connect to it, and the subsequent port also needs to be set to the exposed port.

[error] kmjKafkaJMX$ - Failed to connect to service:jmx:rmi:///jndi/rmi://9.11.8.48:-1/jmxrmi java.lang.IllegalArgumentException: requirement failed: No jmx port but jmx polling enabled!

When viewing topic in the container, the following error is reported (not just the topic command, it seems that all commands will go wrong)

$ bin/kafka-topics.sh --list --zookeeper zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1
# The following is an errorError: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7203; nested exception is:
    java.net.BindException: Address already in use

Solution:

Add unset JMX_PORT; command before the command, and the above command will be transformed into:

$ unset JMX_PORT;bin/kafka-topics.sh --list --zookeeper zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1

Appendix: Common Docker instructions

# View all docker images
# View all running containers docker ps
# View all containers docker ps -a
# Get all container ips
$ docker inspect --format='{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq)
# View the internal logs of the container $ docker logs -f <container ID>
# Enter the container $ docker exec -it <container ID> /bin/basj
# Create a container -d represents background startup docker run --name <container name> -e <parameter> -v <mount data volume> <container ID>
# Restart the container docker restart <container ID>
#Shut down the container docker stop <container id>
# Run the container docker start <container id>

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.

You may also be interested in:

Install Zookeeper under Docker (standalone and cluster)
How to build a Zookeeper cluster with docker

<<: Summary of the characteristics of SQL mode in MySQL

>>: In-depth analysis of JDBC and MySQL temporary tablespace

base target="" specifies the target of the base link to open the frame

Example analysis of mysql stored procedures that trigger error conditions in stored procedures (SIGNAL and RESIGNAL statements)

This article uses examples to illustrate the erro...

Implementation of Docker to build Zookeeper&Kafka cluster

base target="" specifies the target of the base link to open the frame

Implementation code of short video (douyin) watermark removal tool

MySQL full-text search Chinese solution and example code

Three ways to avoid duplicate insertion of data in MySql

HTML+Sass implements HambergurMenu (hamburger menu)

Graphic tutorial on installing MySQL database and configuring environment variables on Mac

How does Vue solve the cross-domain problem of axios request front end

JavaScript implements mouse control of free moving window

Detailed explanation of how to enable https service in Apache under Linux environment

How to use jconsole to monitor remote Tomcat services

Recommend

The whole process record of vue3 recursive component encapsulation

Eclipse configures Tomcat and Tomcat has invalid port solution

Talking about the use of CSS3 custom variables in projects from a project reconstruction

vue-cropper plug-in realizes the encapsulation of image capture and upload component

What are the core modules of node.js

Tutorial on how to quickly deploy clickhouse using docker-compose

Various transformation effects of HTML web page switching

VMware kali virtual machine environment configuration method

Why should the number of rows in a single MySQL table not exceed 5 million?

Detailed explanation of Vue development website SEO optimization method

Some notes on mysql self-join deduplication

Vue implements seamless scrolling of lists

Tomcat Server Getting Started Super Detailed Tutorial

Summary of CSS3 practical methods (recommended)

Example analysis of mysql stored procedures that trigger error conditions in stored procedures (SIGNAL and RESIGNAL statements)