Implementation of Docker to build Zookeeper&Kafka cluster

Implementation of Docker to build Zookeeper&Kafka cluster

I've been learning Kafka recently. When I was preparing to test the cluster status, I felt that it was too troublesome to open three virtual machines or three different port numbers in one virtual machine (well... mainly because I was lazy).

Environment Preparation

A computer with internet access and a CentOS7 virtual machine

Why Use Virtual Machines? Because I use a laptop, the IP changes every time I connect to the network, and I have to modify the configuration file all the time, which is too cumbersome and inconvenient for testing. (This problem can be avoided by using Docker virtual network, but I didn’t know it during the experiment)

Docker Installation

If you have already installed Docker, please ignore this step.

  • Docker supports the following CentOS versions:
  • CentOS 7 (64-bit): The system must be 64-bit and the kernel version must be 3.10 or later.
  • CentOS 6.5 (64-bit) or later: The system must be 64-bit and the kernel version must be 2.6.32-431 or later.
  • CentOS only supports Docker in the kernel included in the distribution.

yum installation

Docker requires the CentOS kernel version to be higher than 3.10. Check the prerequisites above to verify whether your CentOS version supports Docker.

# Check the kernel version $ uname -a
#Install Docker
$ yum -y install docker
#Start the Docker background service$ service docker start
# Since there is no hello-world image locally, a hello-world image will be downloaded and run in the container.
$ docker run hello-world

Script installation

Log in to CentOS using sudo or root privileges.

Make sure the yum package is updated to the latest version.

$ sudo yum update

Get and execute the Docker installation script.

$ curl -fsSL https://get.docker.com -o get-docker.sh
# Executing this script will add the docker.repo repository and install Docker.
$ sudo sh get-docker.sh

Start Docker

$ sudo systemctl start docker
# Verify that Docker is installed successfully and execute a test image in the container.
$ sudo docker run hello-world
$ docker ps

Mirror Acceleration

When I was first asked to configure a domestic mirror source, I refused. However, after using it, I found that the download speed increased duang~ . Therefore, it is strongly recommended that you configure a domestic mirror source.

Open/create the /etc/docker/daemon.json file and add the following content:

{
 "registry-mirrors": ["http://hub-mirror.c.163.com"]
}

Zookeeper cluster construction

Zookeeper image: zookeeper:3.4

Image preparation

$ docker pull zookeeper:3.4

To find images, go to https://hub.docker.com/

docker pull images:TAG // represents pulling the image image of TAG version

Create a separate Zookeeper container

We first create a separate Zookeeper node in the simplest way, and then we create other nodes based on this example.

$ docker run --name zookeeper -p 2181:2181 -d zookeeper:3.4

By default, the configuration file in the container is /conf/zoo.cfg , and the data and log directories are /data and /datalog by default. If necessary, the above directories can be mapped to the host.

Parameter Explanation

--name: Specify the container name
-p: Assign port numbers to ports exposed by the container
-d: Run the container in the background and print the container ID

Cluster construction

The creation method of Zookeeper containers for other nodes is similar to that of independent containers. It should be noted that the node id and the configuration of multiple nodes in the modification file should be specified separately. The corresponding creation commands are as follows:

Create a new docker network

$ docker network create zoo_kafka
$ docker network ls

Zookeeper Container 1

$ docker run -d \
   --restart=always \
   -v /opt/docker/zookeeper/zoo1/data:/data \
   -v /opt/docker/zookeeper/zoo1/datalog:/datalog \
   -e ZOO_MY_ID=1 \
   -p 2181:2181 \
   -e ZOO_SERVERS="server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888" \
   --name=zoo1 \
   --net=viemall-zookeeper \
   --privileged \
   zookeeper:3.4

Zookeeper Container 2

$ docker run -d \
   --restart=always \
   -v /opt/docker/zookeeper/zoo2/data:/data \
   -v /opt/docker/zookeeper/zoo2/datalog:/datalog \
   -e ZOO_MY_ID=2 \
   -p 2182:2181 \
   -e ZOO_SERVERS="server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888" \
   --name=zoo2 \
   --net=viemall-zookeeper \
   --privileged \
   zookeeper:3.4

Zookeeper Container 3

$ docker run -d \
   --restart=always \
   -v /opt/docker/zookeeper/zoo3/data:/data \
   -v /opt/docker/zookeeper/zoo3/datalog:/datalog \
   -e ZOO_MY_ID=3 \
   -p 2183:2181 \
   -e ZOO_SERVERS="server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888" \
   --name=zoo3 \
   --net=viemall-zookeeper \
   --privileged \
   zookeeper:3.4

Although this method also achieves what we want, the steps are too cumbersome and difficult to maintain (late stage of laziness), so we use docker-compose to achieve it.

Docker-compose builds a zookeeper cluster

Create a new docker network

$ docker network create viemall-zookeeper
$ docker network ls

Write the docker-compose.yml script

Directions:

Install docker-compose

# Get the script$ curl -L https://github.com/docker/compose/releases/download/1.25.0-rc2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
# Grant execution permissions $chmod +x /usr/local/bin/docker-compose

Create a new docker-compose.yml file in any directory and copy the following content

Execute the command docker-compose up -d

Command comparison
|Command|Explanation|
|-|-|
|docker-compose up|Start all containers|
|docker-compose up -d|Start and run all containers in the background|
|docker-compose up --no-recreate -d|Do not recreate stopped containers|
|docker-compose up -d test2|Start only the test2 container|
|docker-compose stop|Stop the container|
|docker-compose start|Start the container|
|docker-compose down|Stop and destroy the container|

docker-compose.yml download address: https://github.com/JacianLiu/docker-compose/tree/master/zookeeper
docker-compose.yml details

version: '2'
services:
 zoo1:
  image: zookeeper:3.4 # Image name restart: always # Automatically restart when an error occurs hostname: zoo1
  container_name: zoo1
  privileged: true
  ports: #Port - 2181:2181
  volumes: #Mount data volume - ./zoo1/data:/data
   - ./zoo1/datalog:/datalog 
  environment:
   TZ: Asia/Shanghai
   ZOO_MY_ID: 1 # Node ID
   ZOO_PORT: 2181 # zookeeper port number ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 # zookeeper node list networks:
   default:
    ipv4_address: 172.23.0.11

 zoo2:
  image: zookeeper:3.4
  restart: always
  hostname: zoo2
  container_name: zoo2
  privileged: true
  ports:
   - 2182:2181
  volumes:
   - ./zoo2/data:/data
   - ./zoo2/datalog:/datalog
  environment:
   TZ: Asia/Shanghai
   ZOO_MY_ID: 2
   ZOO_PORT: 2181
   ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
  networks:
   default:
    ipv4_address: 172.23.0.12

 zoo3:
  image: zookeeper:3.4
  restart: always
  hostname: zoo3
  container_name: zoo3
  privileged: true
  ports:
   - 2183:2181
  volumes:
   - ./zoo3/data:/data
   - ./zoo3/datalog:/datalog
  environment:
   TZ: Asia/Shanghai
   ZOO_MY_ID: 3
   ZOO_PORT: 2181
   ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
  networks:
   default:
    ipv4_address: 172.23.0.13

networks:
 default:
  external:
   name: zoo_kafka

verify

From the figure we can see that there is a Leader and two Flower . So far our Zookeeper cluster has been built.

Kafka cluster construction

With the above foundation, is it still a problem to build a Kafka cluster? In fact, it’s just that the values ​​of several variables are different.

With the above example, you don’t need to bother with a single-node Kafka . You can directly use the docker-compose method to deploy three nodes. In fact, the methods are similar. As mentioned above, they are just some different properties. At this time, we don’t need to create a new Docker network. You can directly use the network created when building the Zookeeper cluster!

Environment Preparation

Kafka image: wurstmeister/kafka
Kafka-Manager image: sheepkiller/kafka-manager

# If no version is specified, the latest version of the image will be pulled by default. docker pull wurstmeister/kafka
docker pull sheepkiller/kafka-manager

Write the docker-compose.yml script

Directions:

Install docker-compose

# Get the script$ curl -L https://github.com/docker/compose/releases/download/1.25.0-rc2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
# Grant execution permissions $chmod +x /usr/local/bin/docker-compose

Create a new docker-compose.yml file in any directory and copy the following content

Execute the command docker-compose up -d

Command comparison
|Command|Explanation|
|-|-|-|
|docker-compose up|Start all containers|
|docker-compose up -d|Start and run all containers in the background|
|docker-compose up --no-recreate -d|Do not recreate stopped containers|
|docker-compose up -d test2|Start only the test2 container|
|docker-compose stop|Stop the container|
|docker-compose start|Start the container|
|docker-compose down|Stop and destroy the container|

docker-compose.yml download address: https://github.com/JacianLiu/docker-compose/tree/master/zookeeper
docker-compose.yml details

version: '2'

services:
 broker1:
  image: wurstmeister/kafka
  restart: always
  hostname: broker1
  container_name: broker1
  privileged: true
  ports:
   - "9091:9092"
  environment:
   KAFKA_BROKER_ID: 1
   KAFKA_LISTENERS: PLAINTEXT://broker1:9092
   KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker1:9092
   KAFKA_ADVERTISED_HOST_NAME: broker1
   KAFKA_ADVERTISED_PORT: 9092
   KAFKA_ZOOKEEPER_CONNECT: zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1
   JMX_PORT: 9988
  volumes:
   - /var/run/docker.sock:/var/run/docker.sock
   - ./broker1:/kafka/kafka\-logs\-broker1
  external_links:
  - zoo1
  - zoo2
  - zoo3
  networks:
   default:
    ipv4_address: 172.23.0.14

 broker2:
  image: wurstmeister/kafka
  restart: always
  hostname: broker2
  container_name: broker2
  privileged: true
  ports:
   - "9092:9092"
  environment:
   KAFKA_BROKER_ID: 2
   KAFKA_LISTENERS: PLAINTEXT://broker2:9092
   KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker2:9092
   KAFKA_ADVERTISED_HOST_NAME: broker2
   KAFKA_ADVERTISED_PORT: 9092
   KAFKA_ZOOKEEPER_CONNECT: zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1
   JMX_PORT: 9988
  volumes:
   - /var/run/docker.sock:/var/run/docker.sock
   - ./broker2:/kafka/kafka\-logs\-broker2
  external_links: #Connect to containers outside this compose file
  - zoo1
  - zoo2
  - zoo3
  networks:
   default:
    ipv4_address: 172.23.0.15

 broker3:
  image: wurstmeister/kafka
  restart: always
  hostname: broker3
  container_name: broker3
  privileged: true
  ports:
   - "9093:9092"
  environment:
   KAFKA_BROKER_ID: 3
   KAFKA_LISTENERS: PLAINTEXT://broker3:9092
   KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker3:9092
   KAFKA_ADVERTISED_HOST_NAME: broker3
   KAFKA_ADVERTISED_PORT: 9092
   KAFKA_ZOOKEEPER_CONNECT: zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1
   JMX_PORT: 9988
  volumes:
   - /var/run/docker.sock:/var/run/docker.sock
   - ./broker3:/kafka/kafka\-logs\-broker3
  external_links: #Connect to containers outside this compose file
  - zoo1
  - zoo2
  - zoo3
  networks:
   default:
    ipv4_address: 172.23.0.16

 kafka-manager:
  image: sheepkiller/kafka-manager:latest
  restart: always
  container_name: kafka-manager
  hostname: kafka-manager
  ports:
   - "9000:9000"
  links: # Connect to the container created by this compose file
   - broker1
   - broker2
   - broker3
  external_links: #Connect to containers outside this compose file
   - zoo1
   - zoo2
   - zoo3
  environment:
   ZK_HOSTS: zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1
   KAFKA_BROKERS: broker1:9092,broker2:9092,broker3:9092
   APPLICATION_SECRET: letmein
   KM_ARGS: -Djava.net.preferIPv4Stack=true
  networks:
   default:
    ipv4_address: 172.23.0.10

networks:
 default:
  external: # Use the created network name: zoo_kafka

verify

We open the management page of kafka-manager , the access path is, host ip:9000;


If shown, fill in the address of the Zookeeper cluster, scroll to the bottom and click save

Click on the cluster you just added, and you can see that there are three nodes in the cluster.

Problems encountered during the construction process

Mounting the data volume causes an infinite restart. Checking log prompts: chown: changing ownership of '/var/lib/mysql/....': Permission denied

Solution:

  • Add --privileged=true to docker run to give the container specific permissions
  • Temporarily disable selinux: setenforce 0
  • Add selinux rules to change the security text of the directory to be mounted

kafka-manager reports jmx related errors,

Solution:

  • Add the environment variable JMX_PORT=port to each kafka node
  • After adding it, I found that I couldn't connect, and it was a network connection problem. So I exposed each jmx port and then let the firewall pass, which solved the problem.
  • It is best to set KAFKA_ADVERTISED_HOST_NAME to the IP address of the host machine. Code or tools outside the host machine can connect to it, and the subsequent port also needs to be set to the exposed port.

[error] kmjKafkaJMX$ - Failed to connect to service:jmx:rmi:///jndi/rmi://9.11.8.48:-1/jmxrmi java.lang.IllegalArgumentException: requirement failed: No jmx port but jmx polling enabled!

When viewing topic in the container, the following error is reported (not just the topic command, it seems that all commands will go wrong)

$ bin/kafka-topics.sh --list --zookeeper zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1
# The following is an errorError: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7203; nested exception is:
    java.net.BindException: Address already in use

Solution:

Add unset JMX_PORT; command before the command, and the above command will be transformed into:

$ unset JMX_PORT;bin/kafka-topics.sh --list --zookeeper zoo1:2181/kafka1,zoo2:2181/kafka1,zoo3:2181/kafka1

Appendix: Common Docker instructions

# View all docker images
# View all running containers docker ps
# View all containers docker ps -a
# Get all container ips
$ docker inspect --format='{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq)
# View the internal logs of the container $ docker logs -f <container ID>
# Enter the container $ docker exec -it <container ID> /bin/basj
# Create a container -d represents background startup docker run --name <container name> -e <parameter> -v <mount data volume> <container ID>
# Restart the container docker restart <container ID>
#Shut down the container docker stop <container id>
# Run the container docker start <container id>

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.

You may also be interested in:
  • Install Zookeeper under Docker (standalone and cluster)
  • How to build a Zookeeper cluster with docker

<<:  Summary of the characteristics of SQL mode in MySQL

>>:  In-depth analysis of JDBC and MySQL temporary tablespace

Recommend

MySQL Server 8.0.3 Installation and Configuration Methods Graphic Tutorial

This document records the installation and config...

A detailed introduction to Linux file permissions

The excellence of Linux lies in its multi-user, m...

Learn javascript iterator

Table of contents Introduction What does an itera...

How to prevent duplicate submission in jquery project

In new projects, axios can prevent duplicate subm...

Detailed explanation of the available environment variables in Docker Compose

Several parts of Compose deal with environment va...

Detailed explanation of the solution to Tomcat's crash when double-clicking startup.bat

As a programmer who has just learned Tomcat, this...

mysql workbench installation and configuration tutorial under centOS

This article shares the MySQL Workbench installat...

Detailed steps for installing Tomcat, MySQL and Redis with Docker

Table of contents Install Tomcat with Docker Use ...

vue-cli introduction and installation

Table of contents 1. Introduction 2. Introduction...

Three ways to achieve text flashing effect in CSS3 Example code

1. Change the transparency to achieve the gradual...

In the interview, I was asked whether select...for update would lock the table or the row.

Table of contents verify: Combined with the examp...

Vue.js manages the encapsulation of background table components

Table of contents Problem Analysis Why encapsulat...

Nginx configuration location matching rules example explanation

The scope of nginx configuration instructions can...

MySQL uses SQL statements to modify table names

In MySQL, you can use the SQL statement rename ta...

MySQL encryption and decryption examples

MySQL encryption and decryption examples Data enc...