In-depth analysis of MySQL 8.0 redo log

Preface

When I first learned about the implementation of MySQL, I always heard keywords such as redo log, WAL (write-ahead logging), and undo log. I learned that redo log is mainly used to achieve transaction persistence. To further understand the redo log, I looked at the relevant code (source code version: MySQL 8.0.12). Here is a brief summary, mainly introducing how the redo log is generated, how it is written to the disk, and how the user is finally notified.

Generation of redo log

During the execution of read-write transactions, redo logs are continuously generated. Applying for data pages, modifying data pages, recording undo logs, etc. will generate redo logs. MySQL splits user transactions into mtrs (mini transactions). When the redo log is first generated, it is recorded in the mtr and is committed along with the submission of the mtr, and finally falls on the hard disk.

Redo log submission

When mtr is submitted, the redo log in mtr will be written to the log buffer of the system variable log_sys. A new feature of MySQL 8.0 is lock-free redo log submission. Before 8.0, each user thread competed for mutexes and wrote to the log buffer serially, thus ensuring that the LSN grew sequentially without gaps. In 8.0, user threads can write to the log buffer concurrently. If a user thread successfully writes to the log buffer and then flushes the log buffer before the LSN it has written to the disk, it may cause other user threads to flush the log buffer before they have completed writing it.

To solve this problem, MySQL 8.0 introduced the Link_buf data structure to avoid log buffer holes. Link_buf is actually a fixed-length array that tracks the writing status of a section of the log buffer like a sliding window, and moves forward as continuous redo logs are written to the log buffer.

The data structure of Link_buf is as follows:

When the user writes the redo log between start_lsn and end_lsn in the log buffer, the corresponding position of Link_buf will be marked, that is, m_link[start_lsn%m_capacity] will be assigned the value of end_lsn-start_lsn.

The process of redo log recording to log buffer is as follows:

1. First, when each user thread writes a redo log, it first obtains the start_lsn and end_lsn of the redo log from the system global atomic variable log_sys.sn according to the length of the redo log. The atomic variable sn can ensure that the start_lsn-end_lsn interval obtained by each thread is continuous without holes;

2. After the user thread applies for the start_lsn-end_lsn interval, it needs to wait until Link_buf is advanced to a position that it can use.

As shown in the figure, start_lsn0-end_lsn0, start_lsn2-end_lsn2, start_lsn3-end_lsn3 are the newly applied LSN intervals of three user threads; the interval corresponding to start_lsn1-end_lsn1 has been marked on link_buf; start_lsn3-end_lsn3 is too far from tail and needs to wait for link_buf to advance before it can be used;

3. After writing to the log buffer, mark the range of start_lsn->end_lsn to link_buf (Note: because link_buf is only marked at the position of start_lsn%capacity, it will not be affected even if end_lsn exceeds (m_tail, m_tail+m_capacity));

4. When the user thread commits a transaction, the event log_sys.writer_event is set to trigger the log_writer thread to write the log from the redo log buffer to the system cache (the log_writer thread itself will also poll link_buf to determine whether a new log has been written);

5. The log_writer thread advances m_tail and writes the log buffer before m_tail to disk.

Redo log storage and notification

The previous section briefly describes how redo logs are submitted. When redo logs are submitted and written to disk, multiple threads are involved. The relationship between them is as follows:

When the user thread commits a read-write transaction, some redo logs are generated and recorded in the redo log buffer as the mtr is committed. The user thread then tries to set writer_event to trigger the log_writer thread to write logs and listens to its own flush_events[i] event.

The log_writer thread advances Link_buf.m_tail, writes the redo log before the maximum continuous LSN into the system cache, and sets the flusher_event to trigger the log_flusher thread;

The log_flusher thread flushes the logs written to the system cache to disk and sets flush_notifier_event to trigger the log_flush_notifier thread to notify the user;

log_flush_notifier calculates the event that needs to be triggered based on the LSN of the flushed disk and notifies the user thread.

In the specific implementation, several member variables in log_sys are used to follow up the writing status of the redo log. log_sys.recent_writtern.m_tail indicates the maximum continuous range of the log buffer; log_sys.write_lsn indicates the location where the data is written to the system cache; and log_sys.flushed_to_disk_lsn indicates the location where the data has been flushed to the disk. The advancement process of each mark is as follows:

Notify user thread

When a user commits a transaction, log_wait_for_write or log_wait_for_flush is called according to the innodb_flush_log_at_trx_commit parameter to wait for the redo log to be written to the system cache or flushed to the hard disk. The notification of user threads is implemented through the log_sys.flush_events event array. In order to avoid too many flush_events being notified at one time, flush_events are divided into different user threads like buckets: redo log is divided into log blocks. Assuming the length of the log_sys.flush_events array is m, the flush of the nth log block is monitored by the flush_events[n%m] event. When the L1th log block to the L2th log block of the log buffer is flushed to disk, the flush_events of the log blocks between L1 and L2 will be set, so that the user threads of the redo log between L1 and L2 will receive the notification.

Summarize

MySQL 8.0 solves the performance impact caused by lock competition when user threads write redo logs by making redo logs lock-free. At the same time, the redo log writing file and redo log flushing are separated from the user thread and extracted into separate threads. The user thread is only responsible for writing the redo log to the log buffer and no longer cares about the details of the redo log flushing to the disk. It only needs to wait for notification from the log_writer thread or the log_flusher thread.

The above is the detailed content of the in-depth analysis of MySQL 8.0 redo log. For more information about MySQL 8.0 redo log, please pay attention to other related articles on 123WORDPRESS.COM!