EBS Voulme and its Snapshot

List of Topics

1) Types of EBS volume
2) What is SSD & HDD
3) Difference between the volumes of EBS
4) What is IOPS/ Provsioned IOPS

5) What is IO Credit and Baseline performance
6) How Throughput Optimized volume (st1) Works
7) What is Baseline and Burstable performance

1) Types of EBS volume

AWS offers following 5 types of volumes, each differs by price performance and some other characteristics. The below listed types are mainly categorized by the two types of Disks first one is SSD and second is HDD.

a) SSD-Genral purpose (gp2)
b) SSD-Provisioned IOPS (io1)
c) HDD-Throughput Optimized (st1)
d) HDD-Cold backed (sc1)
e) Magnetic (Old Generation)

Before proceeding with the types of EBS in Detail will see what is SSD and HDD.

2) What is SSD & HDD

HDD - it will have physical disks and these are mechanical hard drive's. If exposed to elements there may be a data loss. Surface of disk should be free of contaminants. The Head of the disk moves back and front to read the data. HDD - backed volumes are optimized for large streaming workloads where throughput (measured in MiB/s) is a better performance measure than IOPS

SSD - SSD is like a flash Storage. It is too small and light compared to HDD. It has NO moving parts. Life span may be about 2000 years for newer generation and for Older generation is about 72 years if used heavly. Performance is 3 times faster than mechanical hard drive's. And even price is little more compared to HDD. Much reliable compared to HDD. SSD-backed volumes are optimized for transactional workloads involving frequent read/write operations with small I/O size, where the dominant performance attribute is IOPS

Less power 

3) Difference between the volumes of EBS

SSD-Backed Provisioned IOPS (io1) - I/O-intensive, NoSQL and relational databases
SSD-Backed General Purpose (gp2) - Boot volume, low-latency interactive apps, dev & test

HDD-Backed Throughput Optimized (st1) - Big data, data warehouse, log processing

HDD-Backed Cold (sc1) - Colder data requiring fewer scans per day

	General Purpose (gp2)	Provisioned IOPS (io1)	Throughput optimized (st1)	Cold backed (sc1)	Magnetic
Base Disk Type	SSD	SSD	HDD	HDD	Previous generation HDD
Use Cases	Recommended for frequently accessed workloads System boot volumes Virtual desktops Low-latency interactive apps Development and test environments	For Critical business applications that require sustained IOPS performance, or more than 10,000 IOPS or 160 MiB/s of throughput per volume Large database workloads,	Streaming workloads requiring consistent, fast throughput at a low price Big data Data warehouses Log processing Cannot be a boot volume	Throughput-oriented storage for large volumes of data that is infrequently accessed Scenarios where the lowest storage cost is important Cannot be a boot	Recommended for infrequently accessed workloads
Volume Size	1 GiB- 16TiB	4 GiB-16TiB	500 GiB-16TB	500 GiB- 16 TiB	1GiB-1TiB
	10,000	20,000	500	250
	65,000	65,000	65,000	65,000	48,000
Throughput Per instance	1,250 MiB/s	1,250 MiB/s	1,250 MiB/s	1,250 MiB/s	1,250 MiB/s
Performance based on attribute	OPS	OPS	MiB/s	MiB/s	OPS

4) What is IOPS

IOPS means Input/output operations per second, is used to measure performance of Hard disk drives (HDD), solid state drives (SSD), and storage area networks (SAN). IOPS is measured in integer. 

For more understanding of IOPS it can be compared to rpm of a car engine. If a vehicle is in neutral, stating that the engine is capable of spinning at 10,000 rpms in that moment is meaningless. Without taking into account the data block size (or I/O size), read/write activity or I/O stream, IOPS as a stand-alone measurement says little.

IOPS are a unit of measure representing input/output operations per second. The operations are measured in KiB, and the underlying drive technology determines the maximum amount of data that a volume type counts as a single I/O. I/O size is capped at 256 KiB for SSD volumes and 1,024 KiB for HDD volumes because SSD volumes handle small or random I/O much more efficiently than HDD volumes.

When small I/O operations are physically contiguous, Amazon EBS attempts to merge them into a single I/O up to the maximum size. For example, for SSD volumes, a single 1,024 KiB I/O operation counts as 4 operations (1,024÷256=4), while 8 contiguous I/O operations at 32 KiB each count as 1operation (8×32=256). However, 8 random I/O operations at 32 KiB each count as 8 operations. Each I/O operation under 32 KiB counts as 1 operation.

Similarly, for HDD-backed volumes, both a single 1,024 KiB I/O operation and 8 sequential 128 KiB operations would count as one operation. However, 8 random 128 KiB I/O operations would count as 8 operations.

Consequently, when you create an SSD-backed volume supporting 3,000 IOPS (either by provisioning an io1 volume at 3,000 IOPS or by sizing a gp2 volume at 1000 GiB), and you attach it to an EBS-optimized instance that can provide sufficient bandwidth, you can transfer up to 3,000 I/Os of data per second, with throughput determined by I/O size.

For example let’s compare how two workloads accessing the same amount of data require a significantly different amount of IOPS. 

The first workload requires reading ten 750MB files, 7.5GB and it takes 100 seconds for the transfer to occur. This means that the transfer rate is 75MB/s and consumes 10 IOPS, which is well within the capabilities of a single hard disk. The second workload requires reading ten thousand 750KB files, the same amount of data, 7.5GB, but it consumes 10,000 IOPS. Since the typical disk drive can’t generate more than 200 IOPS this request won’t get done in the same 100 seconds.(To Understand transfer of 75 Kg object each 7.5 KG will require less load than transfering 75KG which has 10,000 object.)  This is an example of how different workloads can require significantly different performance, while using the same storage capacity.

5) More About General Purpose volume

The performance of gp2 volumes is tied to volume size, which determines the baseline performance level of the volume and how quickly it accumulates I/O credits; larger volumes have higher baseline performance levels and accumulate I/O credits faster. 

Minimum IOPS - 100
Maximum IOPS - 10,000
Baseline  - 3 1OPS/GiB

Baseline of 3 IOPS per GiB with a minimum of 100 IOPS, burstable to 3000 IOPS. And Maximum IOPS for genral purpose is 10000. Also, IOPS for volume sizes greater than 3333 GiB is 10000 IOPS.

I/O credits represent the available bandwidth that your volume can burst large amounts of I/O when more than the baseline performance is needed. The more credits your volume has for I/O, the more time it can burst beyond its baseline performance.

Each volume receives an initial I/O credit balance of 5.4 million I/O credits, which is enough to sustain the maximum burst performance of 3,000 IOPS for 30 minutes

Volumes larger than 1,000 GiB have a baseline performance that is equal or greater than the maximum burst performance, and their I/O credit balance never changes. The maximum I/O credit balance for a volume is equal to the initial credit balance (5.4 million I/O credits).

Bursting and I/O credits are only relevant to volumes under 1,000 GiB, where burst performance exceeds baseline performance.

The burst duration of a volume is dependent on the size of the volume, the burst IOPS required, and the credit balance when the burst begins. This is shown in the following equation:

                            (Credit balance)

Burst duration  =  ------------------------------------

                   (Burst IOPS) - 3(Volume size in GiB)

Eg: If the credit balance is maximum(5.4 Million) and volume size is 1 GB then the burst duration is

Burst duration = 5400000/3000 - 3*1 = 5400000/2997 = 1801 (in Sec for Minutes 1801/60 = 30), then 30 Minutes is the maximum duration of this burst performance.

Throughput can never exceed 160 MiB/s. The throughput limit for gp2 volumes is 128 MiB/s for volumes less than or equal to 170 GiB and 160 MiB/s for volumes over 170 GiB. When I/O demand drops below the baseline level and unused credits are added to the I/O credit balance, the maximum IOPS performance of the volume will again exceed the baseline. 

6) Provisioned IOPS

Minimum IOPS - 100
Maximum IOPS - 20,000
IOPS max is based on 50:1 Ratio

The throughput limit of io1 volumes is 256 KiB for each IOPS provisioned, up to a maximum of 320 MiB/s (at 1,280 IOPS).

For example, a 100 GiB gp2 volume with an empty credit balance has a baseline performance of 300 IOPS and a throughput limit of 75 MiB/s (300 I/O operations per second * 256 KiB per I/O operation = 75 MiB/s)

6) How Throughput Optimized volume (st1) Works

Subject to throughput and throughput-credit caps, the available throughput of an st1 volume is expressed by the following formula:

(Volume size) x (Credit accumulation rate per TiB) = Throughput

In the above figure in throughput line it is showing Baseline 40 MiB/S  Per TB, as we are creating 500 GB volume the throughput is showing as 20 and 123 is the burstable performance.

For a 1 TiB st1 volume, burst throughput is limited to 250

6) What is Baseline and Busrtable performance

Trying to explain what is baseline performance and burstable performance in laymen way. Baseline performance is something like running a car with normal power and if normal power is not utilized completely, then leftover will be added to nitrous power (which is IO credit bucket) In case if we need more power then we can provision extra power by burning nitrous power till it gets empty (technically using io credit balance at some IOPS rate till the bucket becomes empty) even the nitrous mode will have some max speed like 200 KM/h similarly even provisioned IOPS will be bursted at some maximum IOPS  rate which is 3000 IOPS.

Monitoring the Burst Bucket Balance for gp2, st1, and sc1 Volumes

You can monitor the burst-bucket level for for gp2, st1, and sc1 volumes using the EBS BurstBalance metric available in Amazon CloudWatch. This metric shows the percentage of I/O credits (for gp2) or throughput credits (for st1 and sc1) remaining in the burst bucke

About EBS Snapshot

Snapshots occur asynchronously;

Status of the snapshot is pending until the snapshot is complete (when all of the modified blocks have been transferred to Amazon S3),

you can take a snapshot of a volume while a previous snapshot is in the pending status, having multiple pending snapshots of a volume may result in reduced volume performance until the snapshots complete.

Limit of five pending snapshots for a single gp2, io1, or Magnetic volume, and one pending snapshot for a single st1 or sc1 volume.

Snapshots that are taken from encrypted volumes are automatically encrypted. Volumes that are created from encrypted snapshots are also automatically encrypted.

To create a snapshot for an Amazon EBS volume that serves as a root device, you should stop the instance before taking the snapshot.