View disk IO in Linux and find out the processes that occupy high IO read and write

Background - Online Alert

An online server issued an alarm, and the disk utilization disk.util > 90, and the alarm continued.

After logging in to the server, I used iostat -x 1 10 to view the relevant disk usage information. The relevant screenshots are as follows:

 # If there is no iostat command, use yum install sysstat to install it # iostat -x 1 10 

As can be seen from the above figure, the %util[IO] of the vdb disk is almost 100% due to frequent data reading.

Other Field Descriptions

Device: device name
TPS: The number of IO read and write requests per second. Multiple logical requests can be combined into a single I/O request to the device.
Blk_read/s (kB_read/s, MB_read/s): The amount of data read from the device, expressed in blocks (kilobytes, megabytes) per second. A block is equivalent to a sector, so the block size is 512 bytes.
Blk_wrtn/s (kB_wrtn/s, MB_wrtn/s): The amount of data written to the device, expressed in blocks (kilobytes, megabytes) per second. A block is equivalent to a sector, so the block size is 512 bytes.
Blk_read (kB_read, MB_read): Total number of blocks read (kilobytes, megabytes).
Blk_wrtn (kB_wrtn, MB_wrtn): Total number of written blocks (kilobytes, megabytes).

rrqm/s: The number of read requests merged to the device per second. That is, delta(rmerge)/s
wrqm/s: The number of write requests merged to the device per second. That is, delta(wmerge)/s
r/s: The number of read I/O device reads completed per second. That is delta(rio)/s
w/s: The number of write I/0 devices completed per second. That is, delta(wio)/s
rsec/s (rkB/s, rMB/s): The number of sectors read from the device per second (kilobytes, megabytes). Each sector size is 512 bytes
wsec/s (wkB/s, wMB/s): The number of sectors written to the device per second (kilobytes, megabytes). Each sector size is 512 bytes

avgrq-sz: The average amount of data per device I/O operation (in sectors). That is, delta(rsec+wsec)/delta(rio+wio)
avgqu-sz: average length of the I/O queue sent to the device each time.
await: The average waiting time for each IO request. (including waiting queue time and processing time, in milliseconds)
r_await: average waiting time for each IO read request. (including waiting queue time and processing time, in milliseconds)
w_await: average waiting time for each IO write request. (including waiting queue time and processing time, in milliseconds)
svctm: Average processing time (in milliseconds) for each device I/O operation. warn! Do not trust the value of this field any more; this field will be removed in a future version of sysstat.
%util: What percentage of a second is used for I/O operations, or how much time in a second the I/O queue is non-empty. When this value approaches 100%, device saturation occurs.

Find the process with high IO usage

Through the iotop command

If the command is not available, install it using the yum install iotop command.

# iotop -oP

This command allows you to see more detailed information, such as process number, disk read volume, disk write volume, IO percentage, and the commands involved. "Both grep commands cause large IO read volumes."

Through the pidstat command

# Command meaning: Display I/O statistics, updated once a second # pidstat -d 1 

It can be seen that the grep command occupies a large amount of read IO. Then you can view the relevant process information based on the PID.

Note: The PID in this figure is different from that in the previous figure because the process in the previous figure has been executed, and this figure is the process generated after the execution [both execute the same script].

Summarize

The above is what I introduced to you about how to check disk IO in Linux and find out the processes that occupy a lot of IO read and write. I hope it will be helpful to you!