Realization of real-time file synchronization between Linux servers

Realization of real-time file synchronization between Linux servers

Usage scenarios

For existing servers A and B, if the content in the specified directory of server A (for example, /home/paul/rsync/ ) changes (additions, deletions, modifications, and attribute changes), these changes will be synchronized to the target directory of server B (for example, /home/paul/rsync/ ) in real time.

Data mirroring backup tool Rsync

Rsync is an extremely fast and flexible file copying tool. It supports file copying between the local machine and a remote server. Rsync uses a delta-transfer algorithm, which only needs to transfer the difference between the source and target files, greatly reducing the consumption of network bandwidth and the time spent on copying. Rsync is mostly used for data backup and mirroring.

Rsync uses a fast check algorithm to determine whether a file needs to be synchronized by comparing changes in file size or last modification time.

There are two ways to connect to a remote host using Rsync: using ssh or using rsync daemon. Rsync is used here to implement remote file backup.

Installation and operation of Rsync

Install Rsync

Execute in the terminals of Server A and Server B respectively:

sudo yum install rsync

After the installation is complete, you will find that the rsync configuration file is located in etc/rsyncd.conf. This file is required when daemon is used for synchronization and is not introduced here.

Configure password-free login between servers A and B

Server A executes:

ssh-keygen
ssh-copy-id IP address of server B

Create source and target directories

In Server A:

mkdir /home/paul/rsync

In Server B:

mkdir /home/paul/rsync

Create a test file on Server A

echo "Hello from Server A" >> /home/paul/rsync/demo.txt

Execute file transfer command

On Server A, run:

# (1)
rsync -avPz --progress /home/paul/rsync 192.168.100.130:/home.paul/rsync
# (2)
rsync -avPz --delete --progress /home/paul/rsync 192.168.100.130:/home.paul/rsync

You will find that demo.txt also appears in the /home/paul/rsync directory on server B.

Command analysis:

(1) Copy the files in the /home/paul/rsync directory on server A to /home.paul/rsync on server B (192.168.100.130).

(2) Compare the files on the target side and the files on the source side. If the file on the target side does not exist on the source side, delete the file on the target side.

Problems with Rsync

Rsync is only a file copy tool, it cannot monitor the addition, deletion, and modification operations of source files. After making changes on the source side, you need to execute the rsync command to synchronize the changes to the target side.

Rsync needs to scan the entire directory before each synchronization. If there are many files in the source directory, scanning may take a long time.
In order to meet the requirements of real-time monitoring, we need to introduce another tool: inotify.

File system event monitoring tool inotify

inotify-tools provides a simple interface to inotify. It is a library written in C language and also includes command line tools.

For a detailed introduction to inotify-tools, please click: https://github.com/rvoicilas/inotify-tools/wiki

Installation of inotify-tools

For centos7 system, execute in sequence:

yum install -y epel-release
yum --enablerepo=epel install inotify-tools

Use the inotifywait command to monitor events

The monitoring script is as follows (inotifywait-rsync.sh):

inotifywait -mrq --timefmt '%d/%m/%y %H:%M' --format '%T %w%f' -e modify,delete,create,attrib /home/paul/rsync/ | while read file
do
rsync -avPz --progress /home/paul/rsync/ 192.168.100.130:/home/paul/rsync/
rsync -avPz --delete /home/paul/rsync/ 192.168.100.130:/home/paul/rsync/
echo "${file} was synchronized"
done

Parameter parsing

  • -m keeps monitoring. If this parameter is not set, inotifywait will exit after monitoring one event.
  • -r monitor directories recursively.
  • -q Quiet mode, print less output.
  • --timefmt specifies the output format of time.
  • --format Specifies the format of event output.
  • -e Set the event type to monitor. Here we monitor additions, deletions, modifications and metadata changes.

For each triggered listening time, inotifywait executes the code between do and done. Here, we call the rsync command mentioned earlier to synchronize files.

Add the monitoring script to crontab

crontab -e
* * * * * sh /home/paul/inotifywait-rsync.sh

References
https://rsync.samba.org
https://github.com/rvoicilas/inotify-tools/wiki

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.

You may also be interested in:
  • Linux uses the scp command to copy files to the local computer and copy local files to the remote server
  • How to delete folders, files, and decompress commands on Linux servers
  • How to use Samba to build a shared file service on a Linux server
  • How to upload and download files between Linux server and Windows system
  • Python reads files on Linux server
  • How to upload files and folders to Linux server via SSH
  • How to transfer files between Linux local and server and write commands for uploading and downloading files on Linux server
  • Automatic file synchronization between two Linux servers

<<:  What to do if you forget your Linux/Mac MySQL password

>>:  jQuery implements a simple comment area

Recommend

Detailed explanation of HTML style tags and related CSS references

HTML style tag style tag - Use this tag when decl...

IE8 browser will be fully compatible with Web page standards

<br />According to foreign media reports, in...

VMware, nmap, burpsuite installation tutorial

Table of contents VMware BurpSuite 1. Virtual mac...

How to enable JMX monitoring through Tomcat

Build a simulation environment: Operating system:...

Answers to several high-frequency MySQL interview questions

Preface: In interviews for various technical posi...

Three ways to achieve text flashing effect in CSS3 Example code

1. Change the transparency to achieve the gradual...

Detailed explanation of the solution to image deformation under flex layout

Flex layout is a commonly used layout method nowa...

View the frequently used SQL statements in MySQL (detailed explanation)

#mysql -uroot -p Enter password mysql> show fu...

Docker and portainer configuration methods under Linux

1. Install and use Docer CE This article takes Ce...

HTML sample code for implementing tab switching

Tab switching is also a common technology in proj...

Solve the problem that Docker cannot ping the host machine under Mac

Solution Abandon the Linux virtual machine that c...

Meta declaration annotation steps

Meta declaration annotation steps: 1. Sort out all...

Solve the problem of yum installation error Protected multilib versions

Today, when installing nginx on the cloud server,...

CSS3 sample code to achieve element arc motion

How to use CSS to control the arc movement of ele...