IOPS (Input/Output Operations Per Second, pronounced eye-ops) is a common performance measurement used to benchmarkcomputer storage devices like hard disk drives (HDD), solid state drives (SSD), and storage area networks (SAN).
I had a conversation with someone this last week. And, he asked me a question. The question is that how many disks does one need if you have requirements for 15000 IOPs.
And, I am thinking it depends.
Never had a need to know. And, so this Saturday morning, I decided to take to the web; specifically Google and see if I can find anything on-line.
I am all set, as I have B.o.b playing in the background:
B.o.B – Don’t Let Me Fall
Google brought up very good sites, right away:
- Calculate IOPs in a Storage Array by Scott Rowe
- Calculate IOPs (by Marco Broeken)
There are major faucets to review as you determine your IOPS needs.
- Current Usage Model
- Maximum Throughput Storage Apparatus
The number of IOPS that one actually gets depends on a few things:
- Raid Level
- Ratio of read and write operations
- Balance of Sequential to Random access
- Number of worker threads
- The Queue Depth
- Data Block Sizes
in real world settings other factors can indirectly affect IOPS, including:
- System Setup
- Storage drivers
- OS Background operations
- Network Choice (Fiber Channel [FC] & SCSI)
- Work Load (NTFS, Database)
RAID (redundant array of independent disks, originally redundant array of inexpensive disks is a storage technology that combines multiple disk drive components into a logical unit. Data is distributed across the drives in one of several ways called “RAID levels”, depending on the level of redundancy and performance required.
Marketers representing industry RAID manufacturers later attempted to reinvent the term to describe a redundant array of independent disks as a means of disassociating a low-cost expectation from RAID technology.
Your disk Raid level impacts your IOP counts.
Raid levels should not impact Reads. But, because writes end up being multiplexed (written out unto a few disks to increase recoverability), we pay an expected price during writes.
I am very partial to well written and easy to understand pictorials, and I exceedingly like Scott Rowe’s illustration as to the price one pays for the various Raid Levels:
So from Scott Rowe’s pictorial nakedly copied above, one can see that depending on how much protection you want (in terms of Raid Levels), you end up writing to N Disks or figurative N times.
Reads might actually benefits from RAIDS as multiple disks can be summoned and read from.
We all know that you do not not have straight writes nor reads. So per you calculation, you will probably be asked the ratio of Reads to Writes.
So if you are told ahead of time that you will probably have 80% reads and thus 20% writes, you can see that only 20% of you workload will suffer the consequences of your RAID choice.
So if you have an IOP figure in mind, you can look back and see how the percentile of reads & writes and RAID Level choice will impact the IOPS you will actually get:
In the example listed below:
- % of Reads is 80%
- % of Writes is 20%
- Raid Level of 1 –> Reduces Write by factor of 2
- 10000 IOPS
--percentile of reads: 80% --percentile of writes: 20% --RAID Level - Impact --> (0.8 * IOPS) + ( (0.2 * IOPS) / 2) --> (08.8 * 10000) + (0.2 * 10000) / 2 8000 + 1000 = 9000
so though we shot for 10000 IOPs, we will only 9000 IOPS based on expected workload and RAID Configuration.
In the example listed below:
- % of Reads is 60%
- % of Writes is 40%
- Raid Level of 5 –> Reduces Write by factor of 4
- 15000 IOPS
--percentile of reads: 60% --percentile of writes: 40% --RAID Level - Impact --> (0.6 * IOPS) + ( (0.4 * IOPS) / 4) --> (0.6 * 15000) + (0.4 * 15000) / 4 9000 + (6000) / 4 9000 + 1500 10500
so though we shot for 15,000 IOPs, we will only get 10,500 IOPS based on expected workload and RAID Configuration.
Ratio of Read and Write Operations
We have already covered how reads and writes can impact IOPs. In summary, RAID Levels impacts writes; but reads are not lessened.
Balance of Sequential to Random Access
Sequential reads is much, more faster than Random Access.
The reasons are obvious, outside of SSDs, disks are more likely than not mechanical, rotational apparatus.
The more random the requests are, the more likely the disk heads have to move
- Rotational speed
- Average latency
- Average Speed time
Numerous computational techniques and algorithms have been designed to better coalesce and fetch data.
Will address more on that (in a later write-up).
Number of Worker Threads
Threads are a main-stay of high performant systems. Multi-threaded applications are written to gracefully tackle the various areas of concerns a distributed applications need to address.
Generally speaking, the more worker threads that be can gracefully managed by the system the more responsive the system will appear to be.
Having said that, threads use up resources and inter-process communications between threads have to be accounted for.
Most modern operating systems support Asynchronous I/O. And, how they do so depends on each Application’s architecture and desire\ability to push the Technology boundaries.
Here is a good write up by Bob Dorr:
SQL Server IO Presentation http://blogs.msdn.com/b/psssql/archive/2010/03/24/how-it-works-bob-dorr-s-sql-server-i-o-presentation.aspx
SQL Server also exposes the pending (async) I/O requests in the sys.dm_io_pending_io_requests DMV. I specifically point out that the column ‘io_pending’ is a key to understanding the location of the I/O request and who is responsible for it.
If the value is TRUE it indicates that HasOverlappedIOCompleted returned FALSE and the operating system or driver stack has yet to complete the I/O.
Looking at the io_pending_ms_ticks indicates how long the I/O has been pending. If the column reports FALSE for io_pending it indicates that the I/O has completed at the operating system level and SQL Server is now responsible for handling the remainder of the request.
Using Async I/O means that SQL Server will often exceed the recommended disk queue length depth of (2).
This is by design as I just described with the read ahead logic as one example. SQL Server is more concerned with the number of I/O requests and average disk seconds per transfer than the actual queue depth.
Data Block Sizes
Your data block sizes will vary depending on your Application.
- NTFS – Up to NT 4.0 maximum was 4 KB
- NTFS – Win2K and afterwards different sizes
- MS SQL Server – 8192 bytes or 8K
Aggregated storage like NetApp Filers can have disks carrying different specs – That is some of the disks will have certain rotational speeds, whereas others will have different specs and speeds).
Due to such mixings, it is best to look a bit deep when your are really concerned about disk speeds.
Review Current Usage model
If this is an existing application, review your current usage metrics.
For us it is a Database; specifically MS SQL Server.
To gather usage stats, do the following:
- Use MS Windows Performance Monitor
- Use MS SQL Server \ Management Studio – Activity Monitor – Database I/O Chart
We will use these numbers later, but as a place holder we will call them your needed\desired throughput
What is the maximum you will get from your HBA Choice?
Once HBA is mentioned we are referring to external storage sitting outside of the box.
Using Joshua Townsend’s blog:
Storage Basics – Part IX: Alternate IOPS Formula
For 16 GB FC, we use the same equation:
Megabits/sec MB/s = baud_rate * bytes-actually-used-for-data --------------------- bytes-allocated MegaBytes/sec MB/s = MB/s ------ 8 bits equal 1 byte
We need values for the various “informants”:
- What is the baud rate for 16 GB FC ( https://en.wikipedia.org/wiki/Fibre_Channel ) = 14.025 GB
- Bytes used for data = 64
- Total Data Size = 66
Megabits/sec MB/s = baud_rate (14025) * bytes-actually-used-for-data (62) --------------------- bytes-allocated (64) = 14025 * 62 ------------ 64 = 13586.71875 MegaBytes/sec MB/s = MB/s ------ 8 bits equal 1 byte = 13586.71875 ----------- 8 = 1698.34
That is it!
Our number of 1698.34MB/sec is very close to Joshua Townsend’s number (1700 MB/sec)
What will you get from single disks
In cases where one is using internal or directly attached disks, one needs to look at individual disks capability.
Manufacturer supplies that rating through various performance metrics:
- Sustained Transfer Rate
- Rotational Speed
- Seek time
- Latency time
A combination of these numbers can be used to determine your drive’s capabilities.
Sustained Transfer Rate
Sustained Transfer Rates is published MB/sec. This number probably maps to your Vendor’s requirement and so hopefully you will not have to much work.
Here is data from a couple of collaborating sources:
Please keep in mind these numbers are only for mechanical storage and not SSDs. The numbers for SSDs are so over the top; and they should be covered on their own.
For those curious about SSDs, please reach out to Henry Newmans’s coverage per “SSD IOPS Arms race does it matter”
What is your IOPs requirements
How do you find out what your IOPs needs are?
I will rest on Joshua Townsend’s once again. The specific blog that provided clarity was:
Storage Basics – Part IX: Alternate IOPS Formula
So there we have it… To determine your IOPs
- Get your average requirement for MB/sec
- Determine your block size (set at OS Level or Application Level)
How much IOPs should you ask for
Knowing that your RAID Level will impact your IOPS, the question is now how much IOPS you should ask you SAN Admin for:
Unfortunately, I really struggled with what should be a simple formulae:
After hours, took to the Internet, and Googled for “total iops performance”.
Thanks goodness I did, as I found:
IOPS: Performance Capacity Planning Explained
Array IOPS = (I * r) + (I * w * f)
- r=read percentage (as a decimal point, ie 0.75)
- w=write percentage (as a decimal point, ie 0.25)
- I= Desired IOPS
- f=raid penalty
- RAID-0 = 1
- RAID-1/10 = 2
- RAID-5/50 = 4
- RAID-6/60 = 6
There are some tools for measuring IOPs
- IOMeter (originally developed by Intel) ( http://www.iometer.org/ )
- IOZone ( http://www.iozone.org/ )
- FIO ( http://freecode.com/projects/fio )
References – What is IOPS
References – Calculate IOPS
- Calculate IOPS in a storage array – By Scott Lowe
- Calculate IOPS
- Storage Basics – Part IX : Alternate IOPS Formula
- Calculate Back of Envelope Estimates
References – Storage Capacity Planning
- IOPS: Performance Capacity Planning Explained
References – Raid Levels
- Standard Raid Levels
References – Fibre Channel
- Fibre Channel
- 8 GB – Fibre Channel or 10gb Ethernet with fcoe
References – SSDs Performance
- SSD IOPS Arms race does it matter”
References – NetApp – Mixing Disk Types
- Rules for mixing disk types in aggregates
References – I/O Queue Depth Strategy
- IO Depth Strategy