How to use Maxwell to synchronize MySQL data in real time

How to use Maxwell to synchronize MySQL data in real time

About Maxwell

Maxwell is a daemon written in Java that can read MySQL binlog in real time and write row updates in JSON format to Kafka, RabbitMq, Redis, etc. With MySQL incremental data stream, there are many usage scenarios, such as real-time synchronization of data to cache, synchronization of data to ElasticSearch, data migration, etc.

maxwell official website: http://maxwells-daemon.io
Maxwell source code: https://github.com/zendesk/maxwell

Configuration and use of Maxwell

Maxwell depends on Java SDK, so you need to configure the JDK environment first.

1. Download the Maxwell installation package

root@xxx maxwell]# pwd
/usr/local/maxwell
[root@xxx maxwell]# wget https://github.com/zendesk/maxwell/releases/download/v1.19.5/maxwell-1.19.5.tar.gz
[root@xxx maxwell]# tar zxvf maxwell-1.19.5.tar.gz 
[root@xxx maxwell]# cd maxwell-1.19.5

2. Configure MySQL and open MySQL binlog log

[root@xxx mysql]# vi /usr/local/mysql/my.cnf 

[mysqld]
log-bin=mysql-bin #add this line binlog-format=ROW #select row mode server_id=1 #randomly specify a string that cannot be the same as the name of other machines in the cluster. If there is only one machine, you can specify it at will

Restart the mysql service, log in to mysql, and view the binlog log mode

mysql> show variables like '%log_bin%'
+---------------------------------+-------------------------------+
| Variable_name | Value |
+---------------------------------+-------------------------------+
| log_bin | ON |
| log_bin_basename | /data/mysqldb/mysql-bin |
| log_bin_index | /data/mysqldb/mysql-bin.index |
| log_bin_trust_function_creators | OFF |
| log_bin_use_v1_row_events | OFF |
| sql_log_bin | ON |
+---------------------------------+-------------------------------+
6 rows in set (0.11 sec)

Maxwell needs permission to store state in the database specified by the schema_database option (the default database name is maxwell), so you need to give it permission in advance:

#Create a user yhrepl with synchronized data
mysql> create user 'yhrepl'@'*' identified by 'scgaopan'; 
Query OK, 0 rows affected (0.10 sec)

#This user yhrepl must have the permission to operate the database table that needs to be synchronized mysql> grant all privileges on test.* to 'yhrepl'@'%' identified by 'scgaopan'; 
Query OK, 0 rows affected (0.13 sec)

#Give yhrepl the permission to synchronize datamysql> grant select,replication client,replication slave on *.* to 'yhrepl'@'%' identified by 'scgaopan';
Query OK, 0 rows affected (0.10 sec)
# Maxwell needs permission to store state in the database specified by the schema_database option (the default database name is maxwell)
mysql> grant all privileges on maxwell.* to 'yhrepl'@'%' identified by 'scgaopan';
Query OK, 0 rows affected (0.09 sec)

3. Start Maxwell, mainly introducing the actual combat of writing data to rabbitmq:

[root@xxx maxwell-1.19.5]# vi /usr/local/maxwell/maxwell-1.19.5/config.properties
#Log level log_level=DEBUG

producer=rabbitmq
daemon=true

#Monitored database, mysql user must have permission to read binlog and create new database tables host=47.105.110.xxx
user=yhrepl
password=scgaopan

output_nulls=true
jdbc_options=autoReconnet=true

#Which tables in the database are monitored? filter=exclude: *.*,include: test.AA

#replica_server_id and client_id unique identifier, used for cluster deployment replica_server_id=64
client_id=test-id

#metrics_type=http
#metrics_slf4j_interval=60
#http_port=8111
#http_diagnostic=true # default false

#rabbitmq
rabbitmq_host=47.105.110.xxx
rabbitmq_port=5672
rabbitmq_user=guest
rabbitmq_pass=guest
rabbitmq_virtual_host=/
rabbitmq_exchange=maxwell
rabbitmq_exchange_type=topic
rabbitmq_exchange_durable=false
rabbitmq_exchange_autodelete=false
rabbitmq_routing_key_template=%db%.%table%
rabbitmq_message_persistent=false
rabbitmq_declare_exchange=true

Launch Maxwell:

[root@xxx maxwell-1.19.5]# ./bin/maxwell
#Can be started in the background [root@xxx maxwell-1.19.5]# nohub ./bin/maxwell &

If the startup is successful, the maxwell library will be automatically generated. The library records the status of maxwell synchronization, the ID of the last synchronization, and other information. After the main library fails or the synchronization is abnormal, as long as the maxwell library exists, the next synchronization will be based on the ID of the last synchronization. If the maxwell library is not generated or an error is reported, the mysql user permissions configured in config.properties may be insufficient.

RabbitMQ Operation

RabbitMQ operation, after starting Maxwell, a Maxwell exchange is generated

However, the binding of the corresponding queue, exchange and queue needs to be implemented by the user.

Create a maxwell-test queue:

Bind the queue to the exchange:

Note that the Routing key here is case sensitive.

Modify a record in the database and you can see that there is a record in the maxwell-test queue.

Full synchronization

Use the maxwell-bootstrap command

./bin/maxwell-bootstrap --database xhd --table xhd-sso --host 127.0.0.1 --user xiehd --password xiehd2018 --client_id maxwell_dev

Synchronize all data in the xhd.xhd-sso table and specify maxwell with client_id maxwell_dev to perform synchronization

The previous command is open first, and then start maxwell with client_id=maxwell_dev

./bin/maxwell --client_id maxwell_dev

Wait for the execution to complete

The above is the details of how to use Maxwell to synchronize MySQL data in real time. For more information about using Maxwell to synchronize MySQL data, please pay attention to other related articles on 123WORDPRESS.COM!

You may also be interested in:
  • Python implements a script to synchronize MySQL specified table incremental data to clickhouse
  • Detailed explanation of the actual process of master-slave synchronization of MySQL database
  • Two ways of storing scrapy data in MySQL database (synchronous and asynchronous)
  • Mysql master/slave database synchronization configuration and common errors
  • KTL tool realizes the method of synchronizing data from MySQL to MySQL
  • Tutorial on how to synchronize MySQL data to ElasticSearch using Python
  • Master-slave synchronization configuration and read-write separation of MySQL database
  • Steps to synchronize MongoDB data to MySQL using node.js
  • Master-slave synchronous replication configuration of MySQL database under Linux
  • PHP uses SWOOLE extension to synchronize MySQL data

<<:  The difference between shtml and html

>>:  How to use CSS media query aspect-ratio less

Recommend

Vue.js application performance optimization analysis + solution

Table of contents 1. Introduction 2. Why do we ne...

Node quickly builds the backend implementation steps

1. First install node, express, express-generator...

Example of using store in vue3 to record scroll position

Table of contents Overall Effect Listen for conta...

How to handle super large form examples with Vue+ElementUI

Recently, due to business adjustments in the comp...

MySQL slow query log configuration and usage tutorial

Preface MySQL slow query log is a function that w...

Detailed explanation of the basic functions and usage of MySQL foreign keys

This article uses examples to illustrate the basi...

How to use Docker to package and deploy images locally

First time using docker to package and deploy ima...

Discussion on more reasonable creation rules for MySQL string indexes

Preface Regarding the use of MySQL indexes, we ha...

Tutorial on importing and exporting Docker containers

background The popularity of Docker is closely re...

How to implement Svelte's Defer Transition in Vue

I recently watched Rich Harris's <Rethinki...

Summary of commonly used performance test scripts for VPS servers

Here is a common one-click performance test scrip...

Debian virtual machine created by VirtualBox shares files with Windows host

the term: 1. VM: Virtual Machine step: 1. Downloa...

Summary of Vue first screen performance optimization component knowledge points

Vue first screen performance optimization compone...

How to change apt-get source in Ubuntu 18.04

When using apt-get to install, it will be very sl...