Use PSSH to batch manage Linux servers

pssh is an open source software implemented in Python, used for batch ssh operations on a large number of machines; pssh is a tool that can execute commands on multiple servers and also supports copying files, making it an excellent tool of its kind; I recommend using pssh over the for loop approach! The premise of using pssh is that key authentication access (that is, ssh trust relationship) must be configured on the local machine and other servers.

The following is a record of batch operations using pssh:

1) Install pssh
You can install it directly via yum:

[root@bastion-IDC ~]# yum install -y pssh

2) Usage of pssh

[root@bastion-IDC ~]# pssh --help


-h remote host list file for executing commands
-H user@ip:port file content format [user@]host[:port]
-l Username of the remote machine
-p The maximum number of connections allowed at a time
-o redirect output to a file
-e redirect execution errors to a file
-t sets the timeout for command execution
-A prompts for a password and passes it to ssh (note that this parameter is only a prompt after it is added, you can enter it casually or press Enter without entering it)
-O Set the specific configuration of ssh parameters, refer to the ssh_config configuration file
-x passes multiple SSH commands, multiple commands are separated by spaces and enclosed in quotes
-X Same as -x but only one command can be passed at a time
-i Display standard output and standard error after each host is executed
-I Read each input command and pass it to the ssh process allowing command scripts to be piped to standard input

3) pssh instance description

[root@bastion-IDC ~]# cat hosts.txt //The information format in the list file is "ip:port". If the ssh port used by the local machine and the remote machine is the same, you can omit the port and use the ip directly. However, it is recommended to bring all ports.
192.168.1.101:22
192.168.1.109:22
192.168.1.118:25791
192.168.1.105:25791
As shown above, the four machines are placed in a list file hosts.txt. The local machine has established a trust relationship with these four machines for ssh passwordless login. Note: The machines in the list file must have an ssh trust relationship with the local machine in advance. If not, when pssh is executed in batches, it will not be executed when it is the turn of the machine that has not established a trust relationship.

a) Batch execution of commands

[root@bastion-IDC ~]# pssh -h hosts.txt -l root -i 'uptime'
[1] 16:05:48 [SUCCESS] 192.168.1.105
03:03:25 up 79 days, 13:44, 0 users, load average: 0.04, 0.01, 0.00
[2] 16:05:48 [SUCCESS] 192.168.1.118
03:03:32 up 75 days, 15:27, 4 users, load average: 0.96, 0.74, 0.45
Stderr: Address 192.168.1.118 maps to localhost, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
[3] 16:05:48 [SUCCESS] 192.168.1.109
03:03:25 up 61 days, 21:56, 2 users, load average: 0.02, 0.06, 0.18
Stderr: Address 192.168.1.102 maps to localhost, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
[4] 16:05:48 [SUCCESS] 192.168.1.101
16:03:17 up 35 days, 23:45, 1 user, load average: 0.03, 0.04, 0.01
Stderr: Address 192.168.1.101 maps to localhost, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!

If you add the -A parameter, you will still be prompted to enter a password even if you have set up an ssh trust relationship in advance!

[root@bastion-IDC ~]# pssh -h hosts.txt -l root -i -A 'uptime'
Warning: do not enter your password if anyone else has superuser
privileges or access to your account.
Password: //Note that this parameter is only a reminder after it is added. You can enter anything here or just press Enter without entering anything.
[1] 16:08:25 [SUCCESS] 192.168.1.105
03:06:03 up 79 days, 13:46, 0 users, load average: 0.00, 0.00, 0.00
[2] 16:08:25 [SUCCESS] 192.168.1.109
03:06:03 up 61 days, 21:59, 2 users, load average: 0.00, 0.04, 0.15
Stderr: Address 192.168.1.102 maps to localhost, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
[3] 16:08:25 [SUCCESS] 192.168.1.101
16:05:54 up 35 days, 23:47, 1 user, load average: 0.00, 0.02, 0.00
Stderr: Address 192.168.1.101 maps to localhost, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
[4] 16:08:25 [SUCCESS] 192.168.1.118
03:06:10 up 75 days, 15:29, 4 users, load average: 0.85, 0.78, 0.51
Stderr: Address 192.168.1.118 maps to localhost, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
[root@bastion-IDC ~]# pssh -h hosts.txt -l root -i -t 10 -o /root/pssh.log 'uptime && date'
[1] 17:01:02 [SUCCESS] 192.168.1.109
03:58:33 up 79 days, 5:58, 1 user, load average: 0.00, 0.00, 0.00
Wed Feb 8 03:58:33 EST 2017
[2] 17:01:02 [SUCCESS] 192.168.1.105
03:58:40 up 79 days, 14:39, 1 user, load average: 0.00, 0.00, 0.00
Wed Feb 8 03:58:40 EST 2017
[3] 17:01:02 [SUCCESS] 192.168.1.101
16:58:31 up 36 days, 40 min, 1 user, load average: 0.10, 0.03, 0.01
Wed Feb 8 16:58:31 CST 2017
Stderr: Address 192.168.1.101 maps to localhost, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
[4] 17:01:02 [SUCCESS] 192.168.1.118
03:58:47 up 75 days, 16:22, 3 users, load average: 0.20, 0.21, 0.31
Wed Feb 8 03:58:47 EST 2017
Stderr: Address 192.168.1.118 maps to localhost, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
[root@bastion-IDC ~]# ll /root/pssh.log/
total 16
-rw-r--r--. 1 root root 100 Feb 8 17:01 192.168.1.101
-rw-r--r--. 1 root root 99 Feb 8 17:01 192.168.1.105
-rw-r--r--. 1 root root 99 Feb 8 17:01 192.168.1.109
-rw-r--r--. 1 root root 100 Feb 8 17:01 192.168.1.118

b) Batch upload files or directories (pscp.pssh command)
Batch upload local files /mnt/test.file to the /tmp directory on the remote server:

[root@bastion-IDC ~]# pscp.pssh -l root -h hosts.txt /mnt/test.file /tmp/
[1] 16:18:05 [SUCCESS] 192.168.1.105
[2] 16:18:05 [SUCCESS] 192.168.1.109
[3] 16:18:05 [SUCCESS] 192.168.1.101
[4] 16:18:05 [SUCCESS] 192.168.1.118

Batch upload local files /mnt/test.file, /mnt/aa.file, /mnt/bb.file to the /tmp directory on the remote server:

[root@bastion-IDC ~]# pscp.pssh -l root -h hosts.txt /mnt/test.file /mnt/aa.file /mnt/bb.file /tmp/
[1] 16:22:50 [SUCCESS] 192.168.1.109
[2] 16:22:50 [SUCCESS] 192.168.1.105
[3] 16:22:50 [SUCCESS] 192.168.1.118
[4] 16:22:50 [SUCCESS] 192.168.1.101

or:

[root@bastion-IDC ~]# pscp.pssh -l root -h hosts.txt /mnt/{test.file,aa.file,bb.file} /tmp/
[1] 16:23:44 [SUCCESS] 192.168.1.109
[2] 16:23:44 [SUCCESS] 192.168.1.105
[3] 16:23:44 [SUCCESS] 192.168.1.101
[4] 16:23:44 [SUCCESS] 192.168.1.118

Batch upload the local directory /mnt/zhong to the /tmp directory on the remote server (the -r parameter needs to be added when uploading the directory):

[root@bastion-IDC ~]# pscp.pssh -l root -h hosts.txt -r /mnt/zhong /tmp/
[1] 16:19:36 [SUCCESS] 192.168.1.109
[2] 16:19:36 [SUCCESS] 192.168.1.105
[3] 16:19:36 [SUCCESS] 192.168.1.101
[4] 16:19:36 [SUCCESS] 192.168.1.118

Batch upload local directories /mnt/zhong, /mnt/aa, /mnt/vv to the /tmp directory on the remote server

[root@bastion-IDC ~]# pscp.pssh -l root -h hosts.txt -r /mnt/zhong /mnt/aa /mnt/vv /tmp/
[1] 16:21:02 [SUCCESS] 192.168.1.105
[2] 16:21:02 [SUCCESS] 192.168.1.109
[3] 16:21:02 [SUCCESS] 192.168.1.101
[4] 16:21:02 [SUCCESS] 192.168.1.118

or:

[root@bastion-IDC ~]# pscp.pssh -l root -h hosts.txt -r /mnt/{zhong,aa,vv} /tmp/
[1] 16:22:00 [SUCCESS] 192.168.1.109
[2] 16:22:00 [SUCCESS] 192.168.1.105
[3] 16:22:00 [SUCCESS] 192.168.1.101
[4] 16:22:00 [SUCCESS] 192.168.1.118

c) Batch download files or directories (pslurp command)
Batch download certain files from the server to the local computer. You don't have to worry about duplicate names because pssh has created a directory named after the IP address in the file list to store the downloaded files:

[root@bastion-IDC ~]# pslurp -l root -h hosts.txt /etc/hosts .
[1] 16:32:01 [SUCCESS] 192.168.1.109
[2] 16:32:01 [SUCCESS] 192.168.1.105
[3] 16:32:01 [SUCCESS] 192.168.1.101
[4] 16:32:01 [SUCCESS] 192.168.1.118
[root@bastion-IDC ~]# ll
total 123
drwxr-xr-x. 2 root root 4096 Feb 8 16:32 192.168.1.101
drwxr-xr-x. 2 root root 4096 Feb 8 16:32 192.168.1.105
drwxr-xr-x. 2 root root 4096 Feb 8 16:32 192.168.1.109
drwxr-xr-x. 2 root root 4096 Feb 8 16:32 192.168.1.118
[root@bastion-IDC ~]# ll 192.168.1.101
total 4
-rw-r--r--. 1 root root 224 Feb 8 16:32 hosts
[root@bastion-IDC ~]# ll 192.168.1.109
total 4
-rw-r--r--. 1 root root 252 Feb 8 16:32 hosts
[root@bastion-IDC ~]# ll 192.168.1.105
total 4
-rw-r--r--. 1 root root 252 Feb 8 16:32 hosts
[root@bastion-IDC ~]# ll 192.168.1.118
total 4
-rw-r--r--. 1 root root 212 Feb 8 16:32 hosts

Also pay special attention to:
The batch download operation above can only be downloaded to the local current directory, and cannot be followed by a specified path in the command:

[root@bastion-IDC ~]# pslurp -l root -h hosts.txt /etc/hosts /mnt/
[1] 16:34:14 [FAILURE] 192.168.1.109 Exited with error code 1
[2] 16:34:14 [FAILURE] 192.168.1.105 Exited with error code 1
[3] 16:34:14 [FAILURE] 192.168.1.101 Exited with error code 1
[4] 16:34:14 [FAILURE] 192.168.1.118 Exited with error code 1

If you want to download to the local /mnt directory, the correct way is to switch to the /mnt directory first, and then execute the download command: (the list file must follow the full path)

[root@bastion-IDC ~]# cd /mnt/
[root@bastion-IDC mnt]# pslurp -l root -h /root/hosts.txt /etc/hosts ./
[1] 16:34:34 [SUCCESS] 192.168.1.109
[2] 16:34:34 [SUCCESS] 192.168.1.105
[3] 16:34:34 [SUCCESS] 192.168.1.118
[4] 16:34:34 [SUCCESS] 192.168.1.101
[root@bastion-IDC mnt]# ll
total 16
drwxr-xr-x. 2 root root 4096 Feb 8 16:34 192.168.1.101
drwxr-xr-x. 2 root root 4096 Feb 8 16:34 192.168.1.105
drwxr-xr-x. 2 root root 4096 Feb 8 16:34 192.168.1.109
drwxr-xr-x. 2 root root 4096 Feb 8 16:34 192.168.1.118

The above is for batch downloading files. If you want to batch download directories, you only need to add a -r parameter!

[root@bastion-IDC mnt]# pslurp -l root -h /root/hosts.txt -r /home/ ./
[1] 16:39:05 [SUCCESS] 192.168.1.109
[2] 16:39:05 [SUCCESS] 192.168.1.105
[3] 16:39:05 [SUCCESS] 192.168.1.101
[4] 16:39:05 [SUCCESS] 192.168.1.118
[root@bastion-IDC mnt]# ll 192.168.1.101
total 8
drwxr-xr-x. 6 root root 4096 Feb 8 16:39 home
-rw-r--r--. 1 root root 224 Feb 8 16:38 hosts
[root@bastion-IDC mnt]# ll 192.168.1.*
192.168.1.101:
total 8
drwxr-xr-x. 6 root root 4096 Feb 8 16:39 home
-rw-r--r--. 1 root root 224 Feb 8 16:38 hosts
192.168.1.105:
total 8
drwxr-xr-x. 4 root root 4096 Feb 8 16:39 home
-rw-r--r--. 1 root root 252 Feb 8 16:38 hosts
192.168.1.109:
total 8
drwxr-xr-x. 4 root root 4096 Feb 8 16:39 home
-rw-r--r--. 1 root root 252 Feb 8 16:38 hosts
192.168.1.118:
total 8
drwxr-xr-x. 3 root root 4096 Feb 8 16:39 home
-rw-r--r--. 1 root root 212 Feb 8 16:38 hosts

d) Batch synchronization (prsync command)
Synchronize the files or directories in the local /mnt/test directory to the remote machine's /mnt/test path

[root@bastion-IDC ~]# prsync -l root -h hosts.txt -r /mnt/test/ /mnt/test/
[1] 16:46:41 [SUCCESS] 192.168.1.109
[2] 16:46:41 [SUCCESS] 192.168.1.105
[3] 16:46:41 [SUCCESS] 192.168.1.118
[4] 16:46:41 [SUCCESS] 192.168.1.101

Synchronize the files or directories in the local /mnt/test directory to the remote machine's /mnt path

[root@bastion-IDC ~]# prsync -l root -h hosts.txt -r /mnt/test/ /mnt/
[1] 16:47:40 [SUCCESS] 192.168.1.109
[2] 16:47:40 [SUCCESS] 192.168.1.105
[3] 16:47:45 [SUCCESS] 192.168.1.101
[4] 16:47:46 [SUCCESS] 192.168.1.118

Notice:
The above batch synchronization directory operation is to synchronize the corresponding directory data of the local machine to the remote machine. The redundant files in the directory on the remote machine will also be retained (the redundant files will not be deleted)

Similarly, for batch synchronization file operations, remove the -r parameter.
Note: When synchronizing files, it is actually a complete overwrite, and all files in the corresponding files on the remote machine will be replaced!
as follows:
Synchronize the local /mnt/test/file file content to the remote server /mnt/test/file file

[root@bastion-IDC ~]# prsync -l root -h hosts.txt /mnt/test/file /mnt/test/file
[1] 16:53:54 [SUCCESS] 192.168.1.109
[2] 16:53:54 [SUCCESS] 192.168.1.105
[3] 16:53:54 [SUCCESS] 192.168.1.101
[4] 16:53:54 [SUCCESS] 192.168.1.118
[root@bastion-IDC ~]# prsync -l root -h hosts.txt /mnt/test/file /mnt/aaa
[1] 16:54:03 [SUCCESS] 192.168.1.109
[2] 16:54:03 [SUCCESS] 192.168.1.105
[3] 16:54:03 [SUCCESS] 192.168.1.101
[4] 16:54:04 [SUCCESS] 192.168.1.118

e) Batch kill processes on remote machines (pnuke command)
For example, batch kill nginx processes on remote machines

[root@bastion-IDC ~]# pnuke -h hosts.txt -l root nginx
[1] 17:09:14 [SUCCESS] 192.168.1.109
[2] 17:09:14 [SUCCESS] 192.168.1.105
[3] 17:09:15 [SUCCESS] 192.168.1.118
[4] 17:09:15 [SUCCESS] 192.168.1.101