Realization of real-time file synchronization between Linux servers

Realization of real-time file synchronization between Linux servers

Usage scenarios

For existing servers A and B, if the content in the specified directory of server A (for example, /home/paul/rsync/ ) changes (additions, deletions, modifications, and attribute changes), these changes will be synchronized to the target directory of server B (for example, /home/paul/rsync/ ) in real time.

Data mirroring backup tool Rsync

Rsync is an extremely fast and flexible file copying tool. It supports file copying between the local machine and a remote server. Rsync uses a delta-transfer algorithm, which only needs to transfer the difference between the source and target files, greatly reducing the consumption of network bandwidth and the time spent on copying. Rsync is mostly used for data backup and mirroring.

Rsync uses a fast check algorithm to determine whether a file needs to be synchronized by comparing changes in file size or last modification time.

There are two ways to connect to a remote host using Rsync: using ssh or using rsync daemon. Rsync is used here to implement remote file backup.

Installation and operation of Rsync

Install Rsync

Execute in the terminals of Server A and Server B respectively:

sudo yum install rsync

After the installation is complete, you will find that the rsync configuration file is located in etc/rsyncd.conf. This file is required when daemon is used for synchronization and is not introduced here.

Configure password-free login between servers A and B

Server A executes:

ssh-keygen
ssh-copy-id IP address of server B

Create source and target directories

In Server A:

mkdir /home/paul/rsync

In Server B:

mkdir /home/paul/rsync

Create a test file on Server A

echo "Hello from Server A" >> /home/paul/rsync/demo.txt

Execute file transfer command

On Server A, run:

# (1)
rsync -avPz --progress /home/paul/rsync 192.168.100.130:/home.paul/rsync
# (2)
rsync -avPz --delete --progress /home/paul/rsync 192.168.100.130:/home.paul/rsync

You will find that demo.txt also appears in the /home/paul/rsync directory on server B.

Command analysis:

(1) Copy the files in the /home/paul/rsync directory on server A to /home.paul/rsync on server B (192.168.100.130).

(2) Compare the files on the target side and the files on the source side. If the file on the target side does not exist on the source side, delete the file on the target side.

Problems with Rsync

Rsync is only a file copy tool, it cannot monitor the addition, deletion, and modification operations of source files. After making changes on the source side, you need to execute the rsync command to synchronize the changes to the target side.

Rsync needs to scan the entire directory before each synchronization. If there are many files in the source directory, scanning may take a long time.
In order to meet the requirements of real-time monitoring, we need to introduce another tool: inotify.

File system event monitoring tool inotify

inotify-tools provides a simple interface to inotify. It is a library written in C language and also includes command line tools.

For a detailed introduction to inotify-tools, please click: https://github.com/rvoicilas/inotify-tools/wiki

Installation of inotify-tools

For centos7 system, execute in sequence:

yum install -y epel-release
yum --enablerepo=epel install inotify-tools

Use the inotifywait command to monitor events

The monitoring script is as follows (inotifywait-rsync.sh):

inotifywait -mrq --timefmt '%d/%m/%y %H:%M' --format '%T %w%f' -e modify,delete,create,attrib /home/paul/rsync/ | while read file
do
rsync -avPz --progress /home/paul/rsync/ 192.168.100.130:/home/paul/rsync/
rsync -avPz --delete /home/paul/rsync/ 192.168.100.130:/home/paul/rsync/
echo "${file} was synchronized"
done

Parameter parsing

  • -m keeps monitoring. If this parameter is not set, inotifywait will exit after monitoring one event.
  • -r monitor directories recursively.
  • -q Quiet mode, print less output.
  • --timefmt specifies the output format of time.
  • --format Specifies the format of event output.
  • -e Set the event type to monitor. Here we monitor additions, deletions, modifications and metadata changes.

For each triggered listening time, inotifywait executes the code between do and done. Here, we call the rsync command mentioned earlier to synchronize files.

Add the monitoring script to crontab

crontab -e
* * * * * sh /home/paul/inotifywait-rsync.sh

References
https://rsync.samba.org
https://github.com/rvoicilas/inotify-tools/wiki

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.

You may also be interested in:
  • Linux uses the scp command to copy files to the local computer and copy local files to the remote server
  • How to delete folders, files, and decompress commands on Linux servers
  • How to use Samba to build a shared file service on a Linux server
  • How to upload and download files between Linux server and Windows system
  • Python reads files on Linux server
  • How to upload files and folders to Linux server via SSH
  • How to transfer files between Linux local and server and write commands for uploading and downloading files on Linux server
  • Automatic file synchronization between two Linux servers

<<:  What to do if you forget your Linux/Mac MySQL password

>>:  jQuery implements a simple comment area

Recommend

CSS pseudo-class: empty makes me shine (example code)

Anyone who has read my articles recently knows th...

In-depth explanation of Vue multi-select list component

A Multi-Select is a UI element that lists all opt...

How to implement hot deployment and hot start in Eclipse/tomcat

1. Hot deployment: It means redeploying the entir...

MySQL index principle and usage example analysis

This article uses examples to illustrate the prin...

A brief description of the relationship between k8s and Docker

Recently, the project uses kubernetes (hereinafte...

Use crontab to run the script of executing jar program regularly in centOS6

1. Write a simple Java program public class tests...

MySQL 5.6.22 installation and configuration method graphic tutorial

This tutorial shares the specific code of MySQL5....

How to add vector icons to web font files in web page production

As we all know, there are two types of images in c...

Detailed explanation of the use of vue-resource interceptors

Preface Interceptor In some modern front-end fram...

Native js to realize the upload picture control

This article example shares the specific code of ...

How to use Element in React project

This is my first time using the element framework...

CSS World--Code Practice: Image Alt Information Presentation

Using the <img> element with the default sr...