How to quickly deploy an Elasticsearch cluster using docker

How to quickly deploy an Elasticsearch cluster using docker

This article will use Docker containers (orchestrated using docker-compose ) to quickly deploy Elasticsearch 集群, which can be used in a development environment (single machine multiple instances) or a production environment.

Note that the 6.x version can no longer use the -Epath.config parameter to specify the loading location of the configuration file. The documentation states:

For the archive distributions, the config directory location defaults to $ES_HOME/config . The location of the >config directory can be changed via the ES_PATH_CONF environment variable as follows:
ES_PATH_CONF=/path/to/my/config ./bin/elasticsearch
Alternatively, you can export the ES_PATH_CONF environment variable via the command line or via your shell profile.

That is, it is set by the environment variable ES_PATH_CONF (official document). Students who deploy multiple instances on a single machine and do not use containers should pay more attention.

Preparation

Install docker & docker-compose

Here we promote the use of daocloud to accelerate the installation:

#docker
curl -sSL https://get.daocloud.io/docker | sh

#docker-compose
curl -L \
https://get.daocloud.io/docker/compose/releases/download/1.23.2/docker-compose-`uname -s`-`uname -m` \
> /usr/local/bin/docker-compose

chmod +x /usr/local/bin/docker-compose

#View the installation results docker-compose -v

Data Directory

#Create data/log directory Here we deploy 3 nodes mkdir /opt/elasticsearch/data/{node0,nod1,node2} -p
mkdir /opt/elasticsearch/logs/{node0,nod1,node2} -p
cd /opt/elasticsearch
#Permissions I'm also confused. Giving privileged doesn't work either, so I just use 0777. chmod 0777 data/* -R && chmod 0777 logs/* -R

#Prevent JVM from reporting an error echo vm.max_map_count=262144 >> /etc/sysctl.conf
sysctl -p

docker-compse orchestration service

Create an orchestration file

vim docker-compose.yml

Parameter Description

- cluster.name=elasticsearch-cluster

Cluster name

- node.name=node0
- node.master=true
- node.data=true

Node name, whether it can be used as a master node, and whether it stores data

- bootstrap.memory_lock=true

Lock the physical memory address of the process to avoid swapping (swapped) to improve performance

- http.cors.enabled=true
- http.cors.allow-origin=*

Enable cors to use the Head plugin

- "ES_JAVA_OPTS=-Xms512m -Xmx512m"

JVM memory size configuration

- "discovery.zen.ping.unicast.hosts=elasticsearch_n0,elasticsearch_n1,elasticsearch_n2"
- "discovery.zen.minimum_master_nodes=2"

Since versions after 5.2.1 do not support multicast, you need to manually specify the tcp data exchange address of each node in the cluster for cluster節點發現and failover . The default port is 9300 If other ports are set, they need to be specified separately. Here we directly use container communication, or we can map 9300 of each node to the host and communicate through the network port.

Set failover selected quorum = nodes/2 + 1

Of course, you can also mount your own configuration file. The configuration file of ES image is /usr/share/elasticsearch/config/elasticsearch.yml , which is mounted as follows:

volumes:
 - path/to/local/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro

docker-compose.yml

version: '3'
services:
 elasticsearch_n0:
  image: elasticsearch:6.6.2
  container_name: elasticsearch_n0
  privileged: true
  environment:
   - cluster.name=elasticsearch-cluster
   - node.name=node0
   - node.master=true
   - node.data=true
   - bootstrap.memory_lock=true
   - http.cors.enabled=true
   - http.cors.allow-origin=*
   - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
   - "discovery.zen.ping.unicast.hosts=elasticsearch_n0,elasticsearch_n1,elasticsearch_n2"
   - "discovery.zen.minimum_master_nodes=2"
  ulimits:
   memlock:
    soft: -1
    hard: -1
  volumes:
   - ./data/node0:/usr/share/elasticsearch/data
   - ./logs/node0:/usr/share/elasticsearch/logs
  ports:
   - 9200:9200
 elasticsearch_n1:
  image: elasticsearch:6.6.2
  container_name: elasticsearch_n1
  privileged: true
  environment:
   - cluster.name=elasticsearch-cluster
   - node.name=node1
   - node.master=true
   - node.data=true
   - bootstrap.memory_lock=true
   - http.cors.enabled=true
   - http.cors.allow-origin=*
   - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
   - "discovery.zen.ping.unicast.hosts=elasticsearch_n0,elasticsearch_n1,elasticsearch_n2"
   - "discovery.zen.minimum_master_nodes=2"
  ulimits:
   memlock:
    soft: -1
    hard: -1
  volumes:
   - ./data/node1:/usr/share/elasticsearch/data
   - ./logs/node1:/usr/share/elasticsearch/logs
  ports:
   - 9201:9200
 elasticsearch_n2:
  image: elasticsearch:6.6.2
  container_name: elasticsearch_n2
  privileged: true
  environment:
   - cluster.name=elasticsearch-cluster
   - node.name=node2
   - node.master=true
   - node.data=true
   - bootstrap.memory_lock=true
   - http.cors.enabled=true
   - http.cors.allow-origin=*
   - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
   - "discovery.zen.ping.unicast.hosts=elasticsearch_n0,elasticsearch_n1,elasticsearch_n2"
   - "discovery.zen.minimum_master_nodes=2"
  ulimits:
   memlock:
    soft: -1
    hard: -1
  volumes:
   - ./data/node2:/usr/share/elasticsearch/data
   - ./logs/node2:/usr/share/elasticsearch/logs
  ports:
   - 9202:9200

Here we open the host's 9200/9201/9202 as http服務端口for node0/node1/node2 respectively, and tcp數據傳輸of each instance uses the default 9300 to manage communication through the container.

If multi-machine deployment is required, map the ES transport.tcp.port: 9300 port to the host machine's xxxx port, and fill in the address of each host agent discovery.zen.ping.unicast.hosts :

#For example, one of the hosts is 192.168.1.100
  ...
  - "discovery.zen.ping.unicast.hosts=192.168.1.100:9300,192.168.1.101:9300,192.168.1.102:9300"
  ...
ports:
 ...
 - 9300:9300

Create and start the service

[root@localhost elasticsearch]# docker-compose up -d
[root@localhost elasticsearch]# docker-compose ps
   Name Command State Ports       
--------------------------------------------------------------------------------------------
elasticsearch_n0 /usr/local/bin/docker-entr ... Up 0.0.0.0:9200->9200/tcp, 9300/tcp
elasticsearch_n1 /usr/local/bin/docker-entr ... Up 0.0.0.0:9201->9200/tcp, 9300/tcp
elasticsearch_n2 /usr/local/bin/docker-entr ... Up 0.0.0.0:9202->9200/tcp, 9300/tcp

#Startup failed to view errors [root@localhost elasticsearch]# docker-compose logs
#At most, it is some access rights/JVM vm.max_map_count setting issues

Check the cluster status

192.168.20.6 是我的服務器地址

Visit http://192.168.20.6:9200/_cat/nodes?v to view the cluster status:

ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.25.0.3 36 98 79 3.43 0.88 0.54 mdi * node0
172.25.0.2 48 98 79 3.43 0.88 0.54 mdi - node2
172.25.0.4 42 98 51 3.43 0.88 0.54 mdi - node1

Verify Failover

Check the status through the cluster interface

Simulate the master node going offline, the cluster starts electing a new master node, and migrates and reshards the data.

[root@localhost elasticsearch]# docker-compose stop elasticsearch_n0
Stopping elasticsearch_n0 ... done

Cluster status (note that the original master node is offline after changing the http port). The downed node is still in the cluster and will be removed after waiting for a period of time without recovery.

ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.25.0.2 57 84 5 0.46 0.65 0.50 mdi - node2
172.25.0.4 49 84 5 0.46 0.65 0.50 mdi * node1
172.25.0.3 mdi-node0

Wait for a while

ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.25.0.2 44 84 1 0.10 0.33 0.40 mdi - node2
172.25.0.4 34 84 1 0.10 0.33 0.40 mdi * node1

Restore node node0

[root@localhost elasticsearch]# docker-compose start elasticsearch_n0
Starting elasticsearch_n0 ... done

Wait for a while

ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.25.0.2 52 98 25 0.67 0.43 0.43 mdi - node2
172.25.0.4 43 98 25 0.67 0.43 0.43 mdi * node1
172.25.0.3 40 98 46 0.67 0.43 0.43 mdi - node0

Observe with Head plug-in

git clone git://github.com/mobz/elasticsearch-head.git
cd elasticsearch-head
npm install
npm run start

The cluster status diagram makes it easier to see the process of automatic data migration

1. The normal data of the cluster is safely distributed on 3 nodes

2. Offline node1 master node cluster starts to migrate data

Migrating

Migration Complete

3. Restore node1

Question Note

elasticsearch watermark

After deployment, when creating the index, it was found that some shards were in the Unsigned state. This was due to the elasticsearch watermark: low, high, flood_stage limitations. By default, an alarm will be issued when the hard disk usage rate is higher than 85% . For development, it is better to turn it off manually. The data will be sharded to each node, and production will make its own decisions.

curl -X PUT http://192.168.20.6:9201/_cluster/settings \
-H 'Content-type':'application/json' \
-d '{"transient":{"cluster.routing.allocation.disk.threshold_enabled": false}}'

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.

You may also be interested in:
  • Detailed explanation of ElasticSearch 6.4.0 cluster construction
  • ElasticSearch reasonable allocation of index fragmentation principle
  • Detailed tutorial for installing ElasticSearch:7.8.0 cluster with docker
  • Detailed explanation of Elasticsearches cluster construction and data sharding process

<<:  Detailed explanation of the implementation principle of Vue2.0/3.0 two-way data binding

>>:  Solution to the problem of flash back after entering the password in MySQL database

Recommend

How to modify the root user password in mysql 8.0.16 winx64 and Linux

Please handle basic operations such as connecting...

How to introduce pictures more elegantly in Vue pages

Table of contents Error demonstration By computed...

A simple way to change the password in MySQL 5.7

This is an official screenshot. After MySQL 5.7 i...

Solve the problem of MySQL reporting Invalid default value for ''operate_time'' error

Execute the create table statement in the databas...

How to modify the initial password of MySQL on MAC

Problem description: I bought a Mac and installed...

How to make your own native JavaScript router

Table of contents Preface Introduction JavaScript...

HTML Tutorial: Collection of commonly used HTML tags (4)

These introduced HTML tags do not necessarily ful...

JS ES new feature of variable decoupling assignment

Table of contents 1. Decoupled assignment of arra...

Modularity in Node.js, npm package manager explained

Table of contents The basic concept of modularity...

Analysis of MySQL lock mechanism and usage

This article uses examples to illustrate the MySQ...

Steps for Vue to use Ref to get components across levels

Vue uses Ref to get component instances across le...

How to configure Nginx to distinguish between PC or mobile phone access to different domain names

The new official website is online, but the exper...

A brief discussion of 3 new features worth noting in TypeScript 3.7

Table of contents Preface Optional Chaining Nulli...

CSS screen size adaptive implementation example

To achieve CSS screen size adaptation, we must fi...

A brief discussion on the implementation principle of Vue slot

Table of contents 1. Sample code 2. See the essen...