Putting Amazon’s Provisioned IOPs to the test

Last week Amazon released Provisioned IOPs and we were fortunate to get beta access to this new storage solution.   And being good geeks we put PIOPS through its paces.   Over the last few weeks we’ve run various tests against PIOPS and have the results of our performance tests below.

To date, standard EBS volumes have supported moderate or bursty I/O requirements. A problem encountered when running database workloads on standard EBS was inconsistent service times and variable performance, in other words not much protection against noisy neighbors in your multi-tenant infrastructure.RAIDing EBS volumes helped reduce that variability but never completely got rid of it.   Amazon created Provisioned IOPs with this problem in mind.   With Provisioned IOPs the goal was to create a reliable and consistent storage solution optimized for database workloads.

To make EBS a better solution for database workloads Amazon focused on improving throughput, service times and consistency of the service.  This new EBS solution they are calling Provisioned IOPs.   Provisioned IOPs has the following features.

  1. Dedicated IOPS:  Now you can purchase EBS volumes with dedicated IOPS.  For example if you want dedicated 500 IOPS per volume you can now provision a 100GB volume with 500IOPS.  Or you can provision a 100GB volume with 1000IOPS.   However you may not exceed a ratio of 1/10 for GB to IOPs.  So a 100GB volume with 1000IOPS is allowed because the ratio is 1/10 but not a 50GB volume with 1000IOPs since that ratio would be 1/20.
  2. IOPS scale linearly:  Want more IOPs than the 1000 for one volume?  Then RAID multiple volumes and you can scale your IOPS up to 10,000IOPS with 10 x 1000IOPs volumes.  Your limiting factor will be your NIC card since in the end EBS is a network attached storage technology.
  3. EBS optimized instances:  One of the problems with EBS was that the NIC of your virtual instance was used for both EBS traffic and all your other traffic.   So if you saturated your network link with other traffic your EBS volume performance suffered.   Amazon has been fiddling under the hood and has now what it’s calling EBS optimized instances.  Basically the network interfaces of the virtual instances now have some kind of QOS in place to ensure that EBS traffic from the virtual machine has dedicated bandwidth to EBS storage.   Amazon is sketchy on details but stated that an m1.large has 500MBits/sec link and m1.xlarge and above has 1000MBits/sec.  This means that on an m1.xlarge and above you could theoretically push up to 896Mbits/sec or 112MB/sec of sustained traffic to EBS.

So how do all these changes compare with standard EBS?   We ran some performance tests to find out.

The setup

We setup two servers one using EBS volumes and another PIOPS volumes.  The PIOPS server was setup with both optimized EBS option and with 1000IOPs.   The EBS volumes were set with the standard option and optimized EBS option was not set.   Details are below.

Server Setup

Server Setup
OS: Ubuntu 12.04 LTS (64bit) 
AMI: ami-82fa58eb 
Amazon Region: US-EAST-1 
Amazon instances tested:  m1.large  & m2.4xlarge 
EBS volume setup
Standard EBS Volumes. 10 x 100GB 
PIOPS volumes. 10 x 100GB 1000IOPs provisioned volumes.
Filesystem: XFS 

RAID setup

RAID software:Lvm2 
LVM version: 2.02.66(2) (2010-05-20) 
Library version: 1.02.48 (2010-05-20) 
Driver version:  4.22.0

RAID configuration

vcreate vg-ebs -n lvol0 -i10 –I64 -l 100%VG   # Note the 64KB stripe size.

Testing software and configuration

Test software:  Sysbench 4.12   # http://sysbench.sourceforge.net
Test configuration: Nine tests were run for each type of test-mode: 
(seqwr, seqrd, rndrd, rndwr, rndrw) 


#sysbench --test=fileio --file-total-size=32GB  --file-test-mode=$TEST-MODE   
--file-block-size=16K --max-time=300 --max-requests=10000000 --num-threads=16   
--init-rng=on --file-num=16 --file-extra-flags=direct --file-fsync-freq=0 run

Test results

IOPS (transactions/sec)

Based upon our nine tests the PIOPS volumes performed better on random reads/writes and random reads on both m1.large and m2.4xlarge servers.   For random writes, sequential reads, and sequential writes PIOPS and EBS performed approximately the same. Of note is that on the m2.4xlarge the server was able to push almost 10K IOPS which is what Amazon has advertised with their new PIOPS service.  In other tests we were able to push up random r/w up to 6000 IOPs on an m1.large instance.

Throughput (MB/sec)

The throughput graphs show the same results as the IOPS results except that the units are now MB/sec.   If you multiply the number of IOPs with the block size if 16K you get this graph.   As you can see the MB/sec of the m1.large is about 80MB/sec.  Considering the throughput of an m1.large is caped at 500Mbits/sec or 62.5MB/sec we can see that the on the m1.large reads and writes they cap out at about 55MB/sec, which is about 88% utilization.  The other 12% should be TCP, OS and application overhead.  On the M2.4xlarge that bandwidth cap is raised to 1000Mbits/sec or 125MB/sec and we see that the throughput of random reads and random writes jump to 109 and 112MB/sec respectively. Again this accounts for about 87% and 89% utilization on the respective bandwidth caps.   This shows that the performance of the PIOPS volumes when using EBS optimized volumes is bound by the throughput limits of the EBS-optimized instances.

Service time (msec)

Looking at the service time averages of all the tests we can see that PIOPS volumes performed on par with or better than EBS volumes across the board.  One outlier is the m1.large sequential writes on PIOPS.   Looking at the individual results (shown below) the service time was consistent across all the tests, so not sure if this is a transient error or something else.

Brendan Gregg in the comments below asked for the 95%-tile of the results. They are in the chart below.

Performance consistency

One benefit of PIOPS is the consistency of the performance.  Notice the chart below measuring IOPs.  Notice the performance of PIOPS was the same across each of the nine tests for all the different workloads, where with EBS the performance of random r/w and random read fluctuates wildly between different tests.  This is due to the read performance being impacted by noisy neighbors in the multi-tenant environment.  Amazon fixed this problem by limiting IOPs per volume and implementing dedicated bandwidth for EBS with their EBS optimized instances.  This graph illustrates this nicely.

The next chart also shows how these features improve service time consistency for PIOPS volumes as well.  Notice how PIOPS’s service times are practically the same over each run, where the EBS volumes are all over the map for random r/w and random reads.

 Conclusion

Provisioned IOPs was designed to address the limitations of EBS, the largest of which was performance inconsistencies due to multi-tenancy issues.   Amazon has attempted to address those issues by implementing dedicated IOPs per volume and EBS optimized instances.  These two features based upon our testing seem to have hit the mark to improve EBS performance especially for database workloads.

Our tests show dramatic improvements on the database workloads when compared to standard EBS volumes.   On average we saw a 2.6 times improvement on random R/W IOPs performance on m1.large instances and a respectable 7.5 times improvement on an m2.4xlarge instance improvement over standard EBS volumes (*however I admit the performance of the standard EBS volumes was pretty bad during these tests.  Additional tests are warranted).   Random reads on the PIOPS volumes saw an improvement of 1.6 times for m1.large and 2.3 times improvement on an m2.4xlarge instance.   Service times on PIOPS volumes improved by a factor of 2.7 and 7.9 times on m1.large and m2.4xlarge respectively for random read/writes.   For Random Reads PIOPS was 1.7 and 2.5 times faster for m1.large and m2.4xlarge.

Based upon our tests PIOPS definitely provides much needed and much sought after performance improvements over standard EBS volumes.  I’m glad to see that Amazon has heeded the calls of its customers and developed a persistent storage solution optimized for database workloads.

We are in the process of preparing mysql tests using sysbench’s oltp option.   We will post those results soon.

  • Brendan Gregg

    Thanks for sharing the details.

    The results are interesting, and the consistency looks promising. Although I’m struggling to separate the networking QOS from the PIOPS. Given that there should be 10k IOPS on the table (if I’m reading this correctly: “PIOPS volumes. 10 x 100GB 1000IOPs provisioned volumes”), the results mostly don’t explore this limit, and look instead to be capped by networking QOS. I know this was mentioned in the introduction as the expected limiting factor, but without testing the PIOPS limit directly we aren’t sure it exists – or if is just implemented indirectly via networking QOS. I’d be interested to see this retested with an I/O size of 1 Kbyte, instead of 16 Kbytes, to hit the PIOPS limit directly.

    Average service time is interesting, but I’d be curious to see 99th percentiles, or better still – latency heatmaps for the full distribution. Especially for the sequential write (creation) m1.large test. I’ve spent much time fixing I/O latency outlier issues on the cloud – where an occasional 1000+ ms I/O causes real application issues, but isn’t that visible when averaged along with many other I/O. For example, the 9,701 IOPS result has an average latency of 1.65 ms, but since it includes 2.9M I/O (for the 300s run, if I did that math correctly), then adding a single 1000ms outlier would bring the average up to 1.6503 ms – and be lost in the rounding.

    • http://www.stratalux.com/ Jeremy Przygode

      You are right on the I/O size limitation. Since we tested 16Kbytes (imposed by mysql needs) the number of I/Os will be limited by bandwidth. As you say if we reduce the I/O size down to 4K or 1K then yes we should hit the 10K limit.

      I have a 95%tile from sysbench for the tests. I haved added the results graph to the post for you to look at.

  • http://twitter.com/kevinminnick Kevin Minnick

    What filesystem did you use for these tests?

    • http://www.stratalux.com/ Jeremy Przygode

      We tested on xfs file system.

  • Pingback: SmartCloud

  • Pingback: Oracle Database as a Service in the Amazon Cloud: Now with APEX, Oracle XML DB and your Data Center’s IPs. — munz & more

  • Hervé Werner

    Hello,

    very nice article, well done !
    PIOPS & EBS optimized instances details are very obscure on Amazon website, after reading your article it is quite much clearer to my mind.

    There is just one thing I did not understand : you wrote “This means that on an m1.xlarge and above you could theoretically push
    up to 896Mbits/sec or 112MB/sec of sustained traffic to EBS.”, why this value of 896 ? How did you come to this number ?

    Do you know the maximum throughput we can get inside an instance ?

    Hervé

  • Hervé Werner

    Hello,

    thank you for your article, it is very clear.

    There is just a thing I did not understand, you wrote :
    “This means that on an m1.xlarge and above you could theoretically push
    up to 896Mbits/sec or 112MB/sec of sustained traffic to EBS.”
    How did you come to this number of “896″ ? Why not 1000 ?

    Do you know the maximum network throughput we can (theoretically) get on AWS ?

    Hervé

  • Thorsten von Eicken

    Nice benchmarks results! One issue I see is that it looks like you tested on a 32GB dataset. On an m2.4xl that fits entirely into memory and really begs the question whether you’re measure the linux buffer cache or the backing disk…
    For the performance consistency tests, did you make multiple runs on the same two instances, or did you launch fresh instances?

  • Pingback: Is Amazon PIOPS Really Better than Standard EBS? | Redis & Memcached

  • Ofer

    Very interesting article. Thanks. One important thing not mentioned is cost: for each volume with 1000 PIOPS you shell out $100 monthly vs $0 for EBS. So, in your setup, you’re looking at $1,000/mo extra just for the PIOPS, excluding the 25% increase in storage costs. in total, $1,125/mo for PIOPS vs $100/mo in regular EBS.. that’s more than 10x the cost. Do you think it is worth 10x the cost?

    • Thane Hubbell

      Why do you think it’s extra? Yes you pay 100.00 – but if you don’t use it you are paying .10 /million IO requests. this .10/million goes way if you used the provisioned storage so now your IO costs are predictable.

  • Eugene A

    Factoring the cost in as part of the equation, a bench-mark I would like to see would be how a standard striped EBS raid performs VS 1 iOPS drive, and if it’s still a worthwhile transition to go from a standard EBS raid (0$ incremental cost) to a $100 additional incremental cost of 1 PIOPS EBS vol.

    A benchmark I’d really love to see is to test a an 8 or 10 standard EBS volume RAID vs. 1 2000 iOPS EBS vs a raid of 2 1000 iOPS volumes.

  • Yudhi Widyatama

    I have drastically different results by clearing the disk cache before doing ‘sysbench run’. The command to clear linux disk cache is sync && (echo 3>/proc/vm/drop_caches). I wonder what the results would be if you do the same.

  • Pingback: EBS Standard Vs Provisioned IOPS « Celingest Blog

  • Pingback: Putting Amazon’s Provisioned IOPs to the test | Stratalux : ちゅどん道中記