Skip to main content

Slides: Disk Latency and Other Random Numbers

| @renice@hachyderm.io

Your browser doesn't support the features required by impress.js, so you are presented with a simplified version of this presentation.

For the best experience please use the latest Chrome, Safari or Firefox browser.

Disk Latency
(and Other Random Numbers)
@AlTobey Open Source Mechanic Datastax
Disclaimer
  • I am not a statistician
Disclaimer
This study is not strictly scientific.
  • Drive diversity
  • Run Time
  • Time limitations
  • Hardware Limitations (PCIe lanes, controllers)
Performance Potential
  • micro benchmarks show what the hw/sw *can* do
  • NOT what it will do
  • what it will do is almost never as much as it can
  • this is OK

The Art of Discovering and Choosing Tradeoffs.

A.K.A. Tuning & Benchmarking

         cd /sys/block/sda/queue

         cat rotational

         echo 1 |sudo tee nomerges

         echo 0 |sudo tee read_ahead_kb

         echo noop |sudo tee scheduler

         echo deadline |sudo tee scheduler
      
Outliers
A data point on a graph or in a set of results that is very much bigger or smaller than the next nearest data point.
WD Velociraptor 10000 RPM (SATA)
Seagate Cheetah 15000 RPM (SAS)
FusionIO ioDrive II (PCI-Express)
RAID0 across 15K SAS, 10K SATA, 7.2K SAS, SSD SATA
RAID0 across 15K SAS, 10K SATA, 7.2K SAS, SSD SATA
RAID0 across 15K SAS, 10K SATA, 7.2K SAS, SSD SATA

iostat -x 5

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.72    0.00    1.31   19.93    0.00   76.04

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0.00     0.00  185.60    0.00 95027.20     0.00  1024.00     3.17   17.08   17.08    0.00   2.78  51.54
sdc               0.00     0.00  185.20    0.00 94822.40     0.00  1024.00    82.46  435.46  435.46    0.00   5.40 100.00
sda               0.00     0.00  184.40    0.00 94412.80     0.00  1024.00     0.89    4.83    4.83    0.00   1.02  18.80
sdd               0.00     0.00  186.40    0.00 95436.80     0.00  1024.00     7.46   40.17   40.17    0.00   4.91  91.52
md0               0.00     0.00  742.40    0.00 380108.80    0.00  1024.00     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.58    0.00    1.03   23.29    0.00   74.10

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0.00     0.00  184.00    0.00 94208.00     0.00  1024.00     1.90   10.36   10.36    0.00   2.41  44.40
sdc               0.00     0.00  185.40    0.00 94924.80     0.00  1024.00    89.72  461.72  461.72    0.00   5.39 100.00
sda               0.00     0.00  184.00    0.00 94208.00     0.00  1024.00     0.87    4.77    4.77    0.00   1.03  19.00
sdd               0.00     0.00  184.40    0.00 94209.60     0.00  1021.80     3.78   20.50   20.50    0.00   4.16  76.74
md0               0.00     0.00  730.00    0.00 373556.80    0.00  1023.44     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.77    0.00    1.71    9.70    0.00   85.81

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0.00     0.00  187.60    0.00 96051.20     0.00  1024.00     2.31   12.24   12.24    0.00   2.58  48.32
sdc               0.00     0.00  186.60    0.00 95539.20     0.00  1024.00    79.91  457.97  457.97    0.00   5.36 100.00
sda               0.00     0.00  188.80    0.00 96665.60     0.00  1024.00     0.90    4.74    4.74    0.00   1.02  19.28
sdd               0.00     0.00  187.80    0.00 95848.80     0.00  1020.75     5.81   30.84   30.84    0.00   4.73  88.74
md0               0.00     0.00  755.80    0.00 386664.80    0.00  1023.19     0.00    0.00    0.00    0.00   0.00   0.00

       
How It's Made
Where the magic happens
fio
[global]
rw=randread
blocksize=4096
ioengine=sync
norandommap=1
direct=1
iodepth=1
iodepth_batch=1
iodepth_batch_complete=1
group_reporting=1
ramp_time=5
time_based=1
runtime=600s
randrepeat=0
unlink=0
numjobs=1
nrfiles=4
size=100g
       
181GB of raw data
11G effio/suites/i-c39a1acda-full
22G effio/suites/ip-172-31-12-247-full-2014-09-07
51G effio/suites/zorak-full-2014-09-04
34G effio/suites/moltar-full-2014-09-08
39G effio/suites/zorak-full-fio-2014-09-09
14G effio/suites/zorak-full-raijin-2014-09-10
10G effio/suites/zorak-raid0-hdd-hdd-hdd-ssd
181G