How to Use rsync in Linux

1. Introduction

rsync is a commonly used Linux application for file synchronization.

It can synchronize files between a local computer and a remote computer, or between two local directories (but does not support synchronization between two remote computers). It can also be used as a file copy tool, replacing cp and mv commands.

The r " in its name stands for remote. rsync actually means "remote sync". Unlike other file transfer tools (such as FTP or scp), the most important feature of rsync is that it will check the existing files on the sender and receiver and only transfer the changed parts (the default rule is that the file size or modification time has changed).

2. Installation

If rsync is not installed on the local or remote computer, you can install it using the following command.

# Debian

$ sudo apt-get install rsync

# Red Hat

$ sudo yum install rsync

# Arch Linux

$ sudo pacman -S rsync

Note that both ends of the transfer must have rsync installed.

3. Basic Usage

3.1, -r parameter

When the rsync command is used locally, it can be used as an alternative to cp and mv commands to synchronize the source directory to the target directory.

$ rsync -r source destination

In the above command, -r means recursion, which includes subdirectories. Note that -r is required, otherwise rsync will not run successfully. source directory indicates the source directory, and destination indicates the target directory.

If there are multiple files or directories that need to be synchronized, you can write it as follows.

$ rsync -r source1 source2 destination

In the above command, both source1 and source2 will be synchronized to the destination directory.

3.2, -a parameter

The -a parameter can replace -r . In addition to recursive synchronization, it can also synchronize meta information (such as modification time, permissions, etc.). Since rsync uses file size and modification time by default to determine whether a file needs to be updated, -a is more useful than -r . The following usage is the common way of writing.

$ rsync -a source destination

If the destination directory does not exist, rsync will automatically create it. After executing the above command, the source directory source is completely copied to the target directory destination , forming a directory structure of destination/source .

If you only want to synchronize the contents of the source directory source to the destination directory destination , you need to add a slash after the source directory.

$ rsync -a source/ destination

After the above command is executed, the contents of source directory are copied to the destination directory, and no source subdirectory is created under destination .

3.3, -n parameter

If you are not sure what results will be produced after rsync is executed, you can first use -n or --dry-run parameter to simulate the results of the execution.

$ rsync -anv source/destination

In the above command, the -n parameter simulates the result of command execution and does not actually execute the command. The -v parameter outputs the results to the terminal so that you can see what will be synchronized.

3.4, --delete parameter

By default, rsync only ensures that all the contents of the source directory (except for files that are explicitly excluded) are copied to the destination directory. It does not keep the two directories identical and does not delete files. If you want to make the target directory a mirror copy of the source directory, you must use the --delete parameter, which will delete files that only exist in the target directory and not in the source directory.

$ rsync -av --delete source/ destination

In the above command, the --delete parameter will make destination a mirror of source .

4. Exclude files

4.1. --exclude parameter

Sometimes, we want to exclude certain files or directories during synchronization. In this case, we can use the --exclude parameter to specify the exclude pattern.

$ rsync -av --exclude='*.txt' source/ destination

# or

$ rsync -av --exclude '*.txt' source/ destination

The above command excludes all TXT files.

Note that rsync will synchronize hidden files that begin with a "dot". If you want to exclude hidden files, you can write --exclude=".*" .

If you want to exclude all files in a directory, but do not want to exclude the directory itself, you can write it as follows.

$ rsync -av --exclude 'dir1/*' source/ destination

To specify multiple exclude patterns, use multiple --exclude parameters.

$ rsync -av --exclude 'file1.txt' --exclude 'dir1/*' source/ destination

Multiple exclude patterns can also take advantage of Bash's large extension function, using only one --exclude parameter.

$ rsync -av --exclude={'file1.txt','dir1/*'} source/ destination

If there are many exclude patterns, you can write them to a file, one pattern per line, and then specify this file with --exclude-from parameter.

$ rsync -av --include="*.txt" --exclude='*' source/ destination

4.2. --include parameter

The --include parameter is used to specify the file patterns that must be synchronized and is often used in conjunction with --exclude .

$ rsync -av --include="*.txt" --exclude='*' source/ destination

The above command specifies that all files will be excluded when syncing, but TXT files will be included.

5. Remote Synchronization

5.1 SSH protocol

In addition to supporting synchronization between two local directories, rsync also supports remote synchronization. It can synchronize local content to remote servers.

$ rsync -av source/ username@remote_host:destination

You can also synchronize remote content to local.

$ rsync -av username@remote_host:source/destination

By default, rsync uses SSH for remote login and data transfer.

Since rsync did not use the SSH protocol in the early days, the -e parameter was needed to specify the protocol, which was changed later. Therefore, -e ssh can be omitted below.

$ rsync -av -e ssh source/ user@remote_host:/destination

However, if the ssh command has additional parameters, you must use the -e parameter to specify the SSH command to be executed.

$ rsync -av -e 'ssh -p 2234' source/ user@remote_host:/destination

In the above command, the -e parameter specifies that SSH uses port 2234.

5.2 rsync protocol

In addition to using SSH, if the other server has an rsync daemon installed and running, you can also use the rsync:// protocol (default port 873) for transmission. The specific writing method is to use double colons to separate the server and the target directory :: .

$ rsync -av source/ 192.168.122.32::module/destination

Note that module in the above address is not the actual path name, but a resource name specified by the rsync daemon, which is assigned by the administrator.

If you want to know the list of all modules assigned by the rsync daemon, you can execute the following command.

$ rsync rsync://192.168.122.32

In addition to using double colons, the rsync protocol can also directly use rsync:// protocol to specify the address.

$ rsync -av source/ rsync://192.168.122.32/module/destination

6. Incremental backup

The biggest feature of rsync is that it can perform incremental backup, that is, only the changed files are copied by default.

In addition to directly comparing the source directory with the target directory, rsync also supports the use of a base directory, which synchronizes the changes between the source directory and the base directory to the target directory.

The specific approach is that the first synchronization is a full backup, and all files are synchronized in the base directory. Each subsequent synchronization is an incremental backup, which only synchronizes the changed parts between the source directory and the base directory and saves these parts in a new target directory. This new target directory also contains all files, but in fact, only those files that have changed exist in this directory, and other unchanged files are hard links to the base directory files.

--link-dest parameter is used to specify the base directory for synchronization.

$ rsync -a --delete --link-dest /compare/path /source/path /target/path

In the above command, --link-dest parameter specifies the base directory /compare/path , then the source directory /source/path is compared with the base directory, the changed files are found, and they are copied to the target directory /target/path . Hard links are created for files that have not changed. The first backup of this command is a full backup, and all subsequent backups are incremental backups.

Below is an example script that backs up a user's home directory.

#!/bin/bash

# A script to perform incremental backups using rsync

set -o errexit
set -o nounset
set -o pipefail

readonly SOURCE_DIR="${HOME}"
readonly BACKUP_DIR="/mnt/data/backups"
readonly DATETIME="$(date '+%Y-%m-%d_%H:%M:%S')"
readonly BACKUP_PATH="${BACKUP_DIR}/${DATETIME}"
readonly LATEST_LINK="${BACKUP_DIR}/latest"

mkdir -p "${BACKUP_DIR}"

rsync -av --delete \
  "${SOURCE_DIR}/" \
  --link-dest "${LATEST_LINK}" \
  --exclude=".cache" \
  "${BACKUP_PATH}"

rm -rf "${LATEST_LINK}"
ln -s "${BACKUP_PATH}" "${LATEST_LINK}"

In the above script, each synchronization will generate a new directory ${BACKUP_DIR}/${DATETIME} and point the soft link ${BACKUP_DIR}/latest to this directory. The next time you back up, ${BACKUP_DIR}/latest will be used as the base directory to generate a new backup directory. Finally, point the soft link ${BACKUP_DIR}/latest to the new backup directory.

7. Configuration items

-a and --archive parameters indicate archive mode, which saves all metadata, such as modification time, permissions, owners, etc., and soft links are also synchronized.

The --append parameter specifies that the file continues to be transferred from where it was interrupted last time.

The --append-verify parameter is similar to the --append parameter, but it will verify the file after the transfer is completed. If verification fails, the entire file will be resent.

-b and --backup parameters specify that when deleting or updating a file that already exists in the target directory, the file is renamed and backed up. The default behavior is to delete it. The renaming rule is to add the file suffix specified by the --suffix parameter, the default is ~ .

The --backup-dir parameter specifies the directory where the file backup is stored, for example, --backup-dir=/path/to/backups .

The --bwlimit parameter specifies the bandwidth limit. The default unit is KB/s, for example, --bwlimit=100 .

-c and --checksum parameters change the verification method of rsync . By default, rsync only checks whether the file size and last modification date have changed. If so, it will be retransmitted. After using this parameter, it will decide whether to retransmit by judging the checksum of the file content.

The --delete parameter deletes files that exist only in the target directory and not in the source directory, ensuring that the target directory is a mirror image of the source directory.

The -e parameter specifies the use of the SSH protocol to transfer data.

The --exclude parameter specifies files that are not to be synchronized, such as --exclude="*.iso" .

--exclude-from parameter specifies a local file containing the file patterns to be excluded, with one line for each pattern.

The --existing and --ignore-non-existing parameters indicate that files and directories that do not exist in the target directory will not be synchronized.

The -h parameter specifies human-readable output.

-h and --help parameters return help information.

The -i parameter indicates the detailed information of the file differences between the output source directory and the target directory.

--ignore-existing parameter means that as long as the file already exists in the target directory, it will be skipped and these files will not be synchronized.

The --include parameter specifies the files to be included during synchronization and is generally used in conjunction with --exclude .

--link-dest parameter specifies the base directory for incremental backups.

The -m parameter specifies not to synchronize empty directories.

--max-size parameter sets the maximum file size limit for transfer, for example, no more than 200KB ( --max-size='200k' ).

--min-size parameter sets the minimum file size limit for transfer, for example, no less than 10KB ( --min-size=10k ).

The -n parameter or --dry-run parameter simulates the operation that would be performed without actually executing it. Use with the -v parameter to see what content will be synchronized.

The -P parameter is a combination of --progress and --partial parameters.

--partial parameter allows resuming an interrupted transfer. When this parameter is not used, rsync will delete the files that are interrupted halfway through the transfer; when this parameter is used, the files that are halfway through the transfer will also be synchronized to the target directory, and the interrupted transfer will be resumed during the next synchronization. Generally needs to be used with --append or --append-verify .

--partial-dir parameter specifies that the files transferred halfway are saved in a temporary directory, such as --partial-dir=.rsync-partial . Generally needs to be used with --append or --append-verify .

The --progress parameter indicates display progress.

The -r parameter indicates recursion, that is, including subdirectories.

--remove-source-files parameter means that after the transfer is successful, the sender's files will be deleted.

--size-only parameter means that only files with changed sizes are synchronized, regardless of differences in file modification times.

The --suffix parameter specifies the suffix to be added to the file name when backing up the file name. The default is ~ .

The -u and --update parameters indicate that files with updated modification times in the target directory will be skipped during synchronization, that is, files with updated timestamps will not be synchronized.

The -v parameter indicates output details. -vv means output more detailed information, -vvv means output the most detailed information.

The --version parameter returns the version of rsync.

The -z parameter specifies to compress data during synchronization.

The above is the details of how to use Linux rsync. For more information about Linux rsync, please pay attention to other related articles on 123WORDPRESS.COM!