Comparative Analysis of MySQL Binlog Log Processing Tools

Canal

Positioning: Based on database incremental log analysis, it provides incremental data subscription and consumption, and currently mainly supports MySQL.

principle:

• Canal simulates the interactive protocol of MySQL slave, pretends to be MySQL slave, and sends dump protocol to MySQL master
• MySQL master receives the dump request and starts pushing binary log to slave (canal)
• Canal parses binary log objects (originally byte streams)

The entire parser process can be roughly divided into several steps:

• Connection obtains the location of the last successful analysis (if it is the first time to start, it obtains the initial location or the binlog location of the current database)
• Connection is established, BINLOG_DUMP command occurs
• Mysql starts pushing Binary Log
• The received Binary Log is parsed by Binlog parser and some specific information is added.
• Passing to the EventSink module for data storage is a blocking operation until the storage is successful
• After successful storage, the Binary Log location is recorded regularly

• Data filtering: Supports wildcard filtering mode, table name, field content, etc.
• Data routing/distribution: Solve the 1:n (one parser corresponds to multiple stores)
• Data merging: solving n:1 (multiple parsers correspond to one store)
• Data processing: additional processing before entering the store, such as join
  

Maxwell

Canal is developed by Java and is divided into server and client. It has many derivative applications, stable performance and powerful functions. Canal requires you to write your own client to consume the data parsed by Canal.

The advantage of Maxwell over Canal is that it is easy to use. It directly outputs data changes as JSON strings, and there is no need to write a client.

Databus

Databus is a low-latency change capture system that has become an integral part of the LinkedIn data processing pipeline. Databus addresses the fundamental requirements for reliably capturing, streaming, and processing major data changes. Databus provides the following features:

• Isolation between sources and consumers
• Guaranteed in-order and at-least-once delivery with high availability
• Start consuming from any point in the change stream, including full bootstrapping capabilities for the entire data.
• Partition consumption
• Source consistency preservation

Alibaba Cloud's Data Transmission Service (DTS)

Data Transmission Service (DTS) is a data flow service provided by Alibaba Cloud that supports data interaction between multiple data sources such as RDBMS (relational database), NoSQL, and OLAP. DTS provides a variety of data transmission capabilities such as data migration, real-time data subscription and real-time data synchronization. It can realize multiple business application scenarios such as data migration without service interruption, data disaster recovery in different locations, multi-active in different locations (unitization), cross-border data synchronization, real-time data warehouse, query report diversion, cache update, asynchronous message notification, etc., to help you build a highly secure, scalable and highly available data architecture.

Advantages: Data Transmission Service DTS supports data transmission between multiple data sources such as RDBMS, NoSQL, and OLAP. It provides multiple data transmission methods such as data migration, real-time data subscription and real-time data synchronization. Compared with third-party data flow tools, Data Transmission Service DTS provides more diverse, high-performance, highly secure and reliable transmission links. At the same time, it provides many convenient functions, which greatly facilitates the creation and management of transmission links.

My personal understanding: It is a message queue that will push the sql objects it has packaged to you, and you can create a service to parse these sql objects yourself.

Eliminate the expensive deployment and maintenance costs. DTS is adapted to Alibaba Cloud RDS (online relational database), DRDS and other products, solving subscription high availability issues in scenarios such as Binlog log recycling, master-slave switching, and VPC network switching. At the same time, targeted performance optimization is carried out for RDS. It is recommended for stability, performance and cost considerations.

The above is the detailed content of the comparative analysis of MySQL Binlog log processing tools. For more information about MySQL Binlog log processing tools, please pay attention to other related articles on 123WORDPRESS.COM!