Estimate Disk/Service IOPS and Throughput
Tags: devops, tutorial
Published: 2023-07-12 15:25:15

# Introduction
Sometimes we need to know the current status of the storage usage of a service in order to find any possible bottleneck in storage side. This tutorial describes how to measure current `IOPS` and `Throughput` on your server.

# Max `IOPS` and `Throughput` of storage:

In this section, we want to estimate the maximum `IOPS` and `Throughput` of storage.

## `dd` command

```bash
dd if=/dev/zero of=/tmp/disk_test_dd.file bs=100M count=1 oflag=dsync
```

- By running this command, `dd` will read from `/dev/zero` (a stream of null bytes) and write 100 megabytes of data to the file `/tmp/disk_test_dd.file`. The write operation will be synchronized and physically written to the disk before `dd` exits, thanks to the `oflag=dsync` option. This command is often used to test the disk write performance or to create files filled with null bytes for various purposes.

**Result:**

```bash
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.22367 s, 469 MB/s
```

- As you can see `Throughput` of the disk was `469 MB/s`

## `fio` command (Recommended)

```bash
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test_disk_fio --filename=test_disk_fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
```

- By running this command, you will initiate a `fio` test that performs random read-write I/O operations with a block size of 4 kilobytes, using a 4-gigabyte test file/device. The test will use the `libaio` I/O engine with direct I/O enabled. The I/O depth is set to 64, and the reads-to-writes ratio is 75:25. The test results will provide performance metrics and insights into the disk's I/O capabilities under these conditions.

**Result:**

```bash
test_disk_fio: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=128MiB/s,w=41.0MiB/s][r=32.8k,w=10.7k IOPS][eta 00m:00s]
test_disk_fio: (groupid=0, jobs=1): err= 0: pid=68138: Mon May 15 10:52:23 2023
   read: IOPS=37.1k, BW=145MiB/s (152MB/s)(3070MiB/21170msec)
   bw (  KiB/s): min=105184, max=246976, per=100.00%, avg=148570.26, stdev=22072.40, samples=42
   iops        : min=26296, max=61744, avg=37142.55, stdev=5518.10, samples=42
  write: IOPS=12.4k, BW=48.5MiB/s (50.8MB/s)(1026MiB/21170msec)
   bw (  KiB/s): min=36384, max=82448, per=100.00%, avg=49650.48, stdev=7312.82, samples=42
   iops        : min= 9096, max=20612, avg=12412.60, stdev=1828.21, samples=42
  cpu          : usr=4.81%, sys=32.05%, ctx=72759, majf=0, minf=153
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=785920,262656,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=3070MiB (3219MB), run=21170-21170msec
  WRITE: bw=48.5MiB/s (50.8MB/s), 48.5MiB/s-48.5MiB/s (50.8MB/s-50.8MB/s), io=1026MiB (1076MB), run=21170-21170msec

Disk stats (read/write):
  sda: ios=785117/262419, merge=0/226, ticks=1027296/246228, in_queue=1273668, util=99.60%
```

- As you see `fio` provides more detailed result

- Mean `IOPS` for read was  `37.1k` and for write was `12.4k`

- Mean `Throughput` for read was `152MB/s` and for write was `50.8MB/s`

# Current `IOPS` and `Throughput` of storage:

In this section, we want to estimate current `IOPS` and `Throughput` of storage.

## With `iostat`

```bash
iostat -xdmb 60 1440
```

- By running this command, `iostat` will start monitoring and displaying disk statistics for all disk partitions every 60 seconds. The statistics will include information about disk utilization, wait time, service time, and the timestamp when the statistics were collected. The monitoring will continue for approximately 1440 minutes (around 24 hours).

**Note:**
You can run `iostat -xdmb 60 1440 >> iostat_res.txt` to redirect results to a file.

**Result:**

```bash
05/13/2023 08:22:19 PM
Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
loop0            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     1.60     0.00   0.00   0.00
sda            444.91   84.11  26980.23   2880.85     3.43    63.74   0.76  43.11    0.31    1.74   0.10    60.64    34.25   0.15   8.08
sda1             0.00    0.00      0.09      0.00     0.00     0.00  11.23   0.00    0.22   12.28   0.00    41.62     0.50   0.13   0.00
sda2           444.91   84.11  26980.14   2880.85     3.43    63.74   0.76  43.11    0.31    1.74   0.10    60.64    34.25   0.15   8.08
sdb            897.80  739.67  96125.22  35384.91    41.59   944.47   4.43  56.08    0.02    0.03   0.03   107.07    47.84   0.03   5.62
```

- You can see the detail of each column in [`iostat` Doc](https://linux.die.net/man/1/iostat)
- At `05/13/2023 08:22:19 PM`, `sda` has `444.91` `IOPS` for reading and `84.11` for write. (From `r/s` and `w/s` columns)
- As same, `sda` has `26980.23 kB/s` `Throughput` for reading and `2880.85 kB/s` for write. (From `rkB/s` and `wkB/s` columns)

**Note:**
This is a result of single 60 seconds. If you enter the command same as above, you will see 1440 tables like this after 1 day.

# Analyze Output in `jupyter notebook` [Optional]

In case you have some servers, and you need to analyze disk activity on them, You can use provided Jupyter Notebook to get more insight into the disks on your servers.

You can see and download the Jupyter Notebook from [this link](https://nbviewer.org/github/anvaari/notebook-share/blob/main/iostat_analysis.ipynb)

# Conclusion

With provided commands and corresponding results, you can estimate needs of current service and see whether there is a bottleneck regarding to I/O or not.

# References

- [Calculate `IOPS` in linux](https://www.howtouselinux.com/post/check-disk-iops-in-linux)
- [Understanding I/O](https://scoutapm.com/blog/understanding-disk-i-o-when-should-you-be-worried)
- Also [ChatGPT](chat.openai.com) :)