Implementation of docker-compose deployment of zk+kafka+storm cluster

Cluster Deployment Overview

Docker installation

yum update -y     
yum install -y yum-utils device-mapper-persistent-data lvm2    
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo    
yum install docker-ce docker-ce-cli containerd.io   
systemctl start docker

docker-compose installation

yum install epel-release   
yum install -y python-pip   
pip install --upgrade pip   
pip install docker-compose

Modify the hosts file of each host machine

vim /etc/hosts   
172.22.12.20 datacloud0   
172.22.12.21 datacloud1   
172.22.12.22 datacloud2   
172.22.12.23 datacloud3   
172.22.12.24 datacloud4

Zookeeper Cluster

Deploy the zk cluster on machines 20, 21, and 22, and write docker-compose files respectively

If only one machine in the cluster is operating normally, the cluster fails.

Take one of them as an example:

zoo:   
    image: zookeeper:3.4.14   
    restart: always   
    hostname: zoo1   
    container_name: zoo1   
    ports:   
     - 2181:2181   
     -2888:2888   
     -3888:3888   
    volumes:   
     - "./zoo/data:/data"   
     - "./zoo/datalog:/datalog"   
    environment:   
     ZOO_MY_ID: 1   
     ZOO_SERVERS: server.1=0.0.0.0:2888:3888 server.2=172.22.12.21:2888:3888 server.3=172.22.12.22:2888:3888

"./zoo/data:/data" is the data directory mount, must be configured

Environment variable configuration, ZOO_MY_ID is 1, 2, and 3 respectively, ZOO_SERVERS is configured with the cluster address, and the current machine is 0.0.0.0

Kakfa Cluster

Deploy the kafka cluster on machines 20, 21, and 22, and write docker-compose files for each (they can be written together with the zk docker-compose file)

Take one of them as an example:

kafka:   
    image: wurstmeister/kafka:2.12-2.2.2   
    restart: always   
    hostname: kafka1   
    container_name: kafka1   
    ports:   
     - "9092:9092"   
    environment:   
     KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.22.12.20:9092   
     KAFKA_ADVERTISED_HOST_NAME: 172.22.12.20   
     KAFKA_ADVERTISED_PORT: 9092   
     KAFKA_ZOOKEEPER_CONNECT: 172.22.12.20:2181,172.22.12.21:2181,172.22.12.22:2181   
    volumes:   
     - ./kafka/logs:/kafka

There is not much to note. KAFKA_ZOOKEEPER_CONNECT is correctly configured with the zk cluster address, and ADVERTISED related configuration is for the current container.

influxdb

influxdb:   
    image: influxdb:1.7   
    restart: always   
    container_name: influxdb   
    ports:   
     - "2003:2003"   
     - "8086:8086"   
     - "8089:8089"   
     - "8091:8091"   
    volumes:   
     - "./influxdb:/var/lib/influxdb"   
    environment:   
     - INFLUXDB_GRAPHITE_ENABLED=true  

"./influxdb:/var/lib/influxdb" is the data directory mount and must be configured. INFLUXDB_GRAPHITE_ENABLED enables the graphite function.

Influxdb simple operation:

• docker exec -it influxdb influx
• show databases
• create database XX
• use XX
• INSERT cpu,host=serverA,region=us_west value=0.64
• SELECT "host", "region", "value" FROM "cpu"

Docker-compose related operations

docker-compose [-f <filename>] up -d   
docker-compose [-f <filename>] down

zkui/kafdrop/grafana

On machine 23, deploy these three web interfaces:

zkui:   
    image: maauso/zkui   
    restart: always   
    container_name: zkui   
    ports:   
     -9090:9090   
    environment:   
     ZKLIST: 172.22.12.20:2181,172.22.12.21:2181,172.22.12.22:2181   
   kafdrop:   
    image: obsidiandynamics/kafdrop:latest   
    restart: always   
    container_name: kafdrop   
    ports:   
     - "9000:9000"   
    environment:   
     KAFKA_BROKERCONNECT: 172.22.12.20:9092,172.22.12.20:9092,172.22.12.20:9092   
   grafana:   
    image: grafana/grafana:master   
    restart: always   
    container_name: grafana   
    ports:   
     - "3000:3000"   
    volumes:   
     - "./grafana:/var/lib/grafana" 

Because it is a web page, just configure the relevant service address and start it.

Grafana needs to be configured with a directory mount, mainly to record custom data sources and panel configurations, etc. (chmod -R 777 grafana/ is required here)

Storm Cluster

The storm cluster consists of three parts: nimbus, supervisor, and ui

Nimbus is the master node and supervisor is the slave node. The former sends tasks to zookeeper, and the latter obtains tasks from zookeeper. ui is a web page.

Deploy three nimbus and supervisor nodes on machines 20, 21, and 22 respectively, and it is recommended to create a separate docker-compose script file and execute it after zookeeper is started.

The docker-compose script is roughly as follows:

nimbus:
  image: storm:2.2.0
  container_name: nimbus1
  command: storm nimbus
  restart: always
  hostname: nimbus1
  ports:
   -6627:6627
  volumes:
   - "./storm.yaml:/conf/storm.yaml"
   - "./nimbus/data:/data"
   - "./nimbus/logs:/logs"

 supervisor:
  image: storm:2.2.0
  container_name: supervisor1
  command: storm supervisor
  hostname: supervisor1
  depends_on:
   -nimbus
  links:
   - nimbus:nimbus
  restart: always
  volumes:
   - "./storm.yaml:/conf/storm.yaml"
   - "./supervisor/data:/data"
   - "./supervisor/logs:/logs"

Nimbus, supervisor, and ui all use the same storm image, but the command parameters are different when starting.

The hostname of nimbus and supervisor is specified here because by default, the host when they are registered to zookeeper is the uuid randomly generated by the docker container.

If nimbus and supervisor are restarted, their container uuids will also be reset, which will conflict with the uuids already saved in zookeeper and cause an error.

The solution on the Internet is: 1. After stopping the storm cluster, delete the storm node on the zookeeper and restart the zookeeper+storm cluster; 2. Build a separate stateless zookeeper for the storm cluster

The solution here is to specify the host when nimbus and supervisor are registered, which will not change when restarted

Some environment variables need to be specified when starting. Here, local configuration file mapping is used. The storm.yaml configuration is as follows:

storm.zookeeper.servers: 
 - "172.22.12.20"
 - "172.22.12.21"
 - "172.22.12.22"
nimbus.seeds:
 - "nimbus1"
 - "nimbus2"
 - "nimbus3"
storm.log.dir: "/logs"
storm.local.dir: "/data"
storm.cluster.mode: distributed

storm.zookeeper.servers is the zookeeper address, and the default port is 2181. nimbus.seeds is the nimbus cluster address, where the hostname customized in docker-compose is used.

Storm-ui can start the single instance, and the docker-compose configuration is as follows:

storm-ui:
  image: storm:2.2.0
  container_name: storm-ui
  command: storm ui
  depends_on:
   -nimbus
  links:
   - nimbus:nimbus
  restart: always
  ports:
   - 8080:8080
  volumes:
   - "./storm.yaml:/conf/storm.yaml"
   - "./ui/data:/data"
   - "./ui/logs:/logs" 
  extra_hosts:
   - "nimbus1:172.22.12.20"
   - "nimbus2:172.22.12.21"
   - "nimbus3:172.22.12.22"
   - "supervisor1:172.22.12.20"
   - "supervisor2:172.22.12.21"
   - "supervisor3:172.22.12.22"

Among them, extra_hosts is the customized host mapping after the container is started, so that after the UI page is started, the registered alias host can correspond to the real IP address.

When deploying Docker containers, you often need to map the host's configuration files to the container so that the configuration will not be reset after the container is restarted.

For example, the storm configuration is: - "./storm.yaml:/conf/storm.yaml". However, after mapping, if there is no storm.yaml in the correct format locally, the startup will fail. In addition to downloading the installation package from the official website to get its configuration file, you can first start a temporary container and copy its configuration file to the host.

docker run -d --name nimbus-tmp storm:2.2.0 storm nimbus

docker cp nimbus-tmp:/conf/storm.yaml local folder path/

canal deployment

Before deploying canal, you need to configure the MySQL to which it is connected:

[mysqld]
log-bin=mysql-bin # Enable binlog
binlog-format=ROW # Select ROW mode server_id=1 # Configuring MySQL replaction requires definition, do not repeat the slaveId of canal

The docker-compose configuration of canal is as follows:

canal-server:
  image: canal/canal-server:v1.1.3
  container_name: canal-server
  ports:
   - 11111:11111
  environment:
   - canal.instance.mysql.slaveId=12
   - canal.auto.scan=false
   - canal.destinations=datacloud
   - canal.instance.master.address=10.23.16.32:3307
   - canal.instance.dbUsername=root
   - canal.instance.dbPassword=chinaunicom@dengfy
   - canal.instance.filter.regex=river\\..*
  volumes:
   - ./canal-server/logs/:/admin/canal-server/logs/

The slaveId needs to be different from the one configured in MySQL, destinations is the custom instance name, and canal.instance.filter.regex is the database table filter to be monitored.

Portainer deployment

Portainer is a visual management tool for Docker containers. It can manage the Docker containers of the above machines, view logs, modify startup scripts, stop containers, etc.

First, you need to open the docker api interface for each machine:

1.vim /usr/lib/systemd/system/docker.service
Add -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock to the ExecStart startup item
2. Restart the docker service systemctl daemon-reload systemctl restart docker

Deploy the portainer container on one of the machines:

Portainer:
  image: portainer/portainer:1.24.1
  restart: always
  container_name: portainer_datacloud
  ports:
   - "8000:8000"
   - "9002:9000"
  volumes:
   - "./portainer/data:/data"
   - "/var/run/docker.sock:/var/run/docker.sock"

"/var/run/docker.sock:/var/run/docker.sock" This mapping relationship is used in stand-alone mode. We use remote API calls, so it can be omitted.

After opening ip:9002, register as a user and log in, go to settings>endpoints>Add endpoint>select docker.

Fill in the target ip:2375 in Endpoint URL and add the target ip in Public IP. After submitting, you can see all its container information.

This is the end of this article about the implementation of docker-compose deployment of zk+kafka+storm cluster. For more relevant docker compose deployment cluster content, please search 123WORDPRESS.COM's previous articles or continue to browse the following related articles. I hope everyone will support 123WORDPRESS.COM in the future!