About EC2 and its Suported Storage, Hosts Types, Network Interface and IP Address

List of Topics

1) What is EC2
2) Supported Storage
3) Instance Purchasing options
5) Way to access EC2
6) Instance type and Family
7) General Information
9) Security Group

10) Types of network interfaces
11) Charging

1) What is EC2 - Virtual servers in the cloud is known as instances

2) AMI (Amazon Machine Image) - Pre-configured templates for your instances, like an ISO image. 

3) Supported Storage
Below are the list of storage options of EC2. In the below list S3 is explained seperately in a different post. You can see a Post named S3 in blog home page. Remaining are explained here.

a) Instance Store (Temporary Storage)
b) EBS Store (Permanent Storage) 
c) S3 Store
d) EFS

3) Instance Purchasing Options

1)  On-Demand instances — Pay, by the hour, for the instances that you launch.

2) Reserved Instances — Purchase, at a significant discount, instances that are always available, for a term from one to three years.

3) Scheduled Instances — Purchase instances that are always available on the specified recurring schedule, for a one-year term.

4) Spot instances — Bid on unused instances, which can run as long as they are available and your bid is above the Spot price, at a significant discount.

5) Dedicated hosts — Pay for a physical host that is fully dedicated to running your instances, and bring your existing per-socket, per-core, or per-VM software licenses to reduce costs.

6) Dedicated instances — Pay, by the hour, for instances that run on single-tenant hardware.

AWS EC2 has 2 different type of instances: Fixed Performance Instances(e.g. M3, C3 etc) and Burstable Performance Instances (e.g. T2). Fixed Performance Instances provides a consistent CPU performance whereas Burstable Performance Instances provide a baseline CPU performance under normal workload. But when the workload increases Burstable Performance Instances have the ability to burst, i.e. increase the CPU performance.

5) How to Access EC2 - AWS CLI, EC2 CLI and AWS Tools

6) Instance types & Family

EC2 instances are mainly classified based on below three resources

a) CPU
b) Memory
c) Network

Instance Family

a) General Purpose          - T2, M4, M3
b) Compute Optimized     - C4, C3
c) Memory Optimized       - R3, X1
d) Accelerated Optimized - C2, G2
e) Storage Optimized        - I2 
f) GPU Enabled                - G2, P2

6) Resource Location

a) Regional resources - AMI, EIP, Security Group and EBS Snapshot
b) AZ                            - EBS Volume and EC2 Instances

7) General Information

a) Static IP address for dynamic Cloud computing is known as Elastic IP

b) When an  Instance is Terminated by default its root EBS volumes are also deleted unless "DeleteOnTermination" is set to false.

c) Each Region is completely independent. Each AZ is completely isolated, but the AZ's are connected through low latency links

d) EC2 Snapshot - Only accessible from EC2 API

e) Reverse DNS for ElP is configurable

Enhance Networking - Supported only in VPC, there is no additional charge for enhanced networking. Need to launch Specific AMI

Reserved Instances

AWS charges 12% of total upfront price for selling Reserved instances. Reserved instances can be migrated between availability zones in the same region.Reservered instances can be used for Auto Scaling

Limits

Number of EIP/Per Region -  5

9) Security Group - An Additional Way of Filtering packet's at AWS level. 

Security group is a virtual firewall that controls the traffic for one or more instances. While launching an instance, you must associate your instance atleast with one security group. Security group can be modified at any time and changes are automatically immediately applied to all instances that are associated with the security group.

By default, security groups allow all outgoing traffic and no Inbound Traffic. You can’t create deny rules with Security group. Security group is specific to VPC. While launching an instance you must specify a security group for that VPC.

Multiple security groups can be assigned to an instance.

10) Types Network Interface's

In Amazon we have two types on network interfaces 1) Primary and 2) ENI which is explained below. 

Primary network interface - Every instance in a VPC has a default network interface, called the primary network interface(eth0). You cannot detach a primary network interface from an instance

Elastic Network Interface (ENI) - An virtual network interface that you can attach to an instance in VPC. It is available for the instance's in VPC only.  If you require a persistent public IP address that can be associated to and from instances as. An security group gets attached to ENI

(i)  Warm attach (When instance is Stopped)

(ii)  Hot attach (When instance is running)

(iii) Cold attach (While instance is being Started)

11) Charging

(i)  On-demand - Payment is based on hourly usage. Partial hour usage will be converted into full hour and billed. Billing cycle will start when the instance starts running and charges will be incurred. Stopping and Not running won't incur any charges. For example if you stop and start multiple times in a single hour then after every stop and start new billing cycle starts so you will be billed multiple times.

(ii)  Reserved -  Reserving option is available for 1 Year and 3 year, and three three types of payment available for reserved instances
    a) Full Upfront -  Full payment while purchasing the reserved instance

    b) Half Upfront - Partial payment While purchasing remaining payment as a part of every month bill.

    c) No upfront - No payment during purchasing and the total amount will be paid as a part of monthly bill

Capacity is guaranteed in reserved instance. At any point of time you will be getting your instance and we wont get any error like capacity utilized. We can resell the instance but there is no refund for the selling of reserved instance.

(iii) Scheduled Reservation - in case of not wanted to, 1 year term define the frequency, lesser than on demand.

(iv) Spot instances  - Spot instances will be started based on the bid pricing. And instances will be terminated with 2 minutes of notice period. Last partial hour is not charged. 

Spot block Option - While submitting the bid if we enable this option AWS won't terminate the instance during that duration irrespective of the price of the instance. Maximum duration for this option is max 6 hours.

(iv) Dedicates hosts  -  

Storage Gateway

What is Storage gateway

Service to connect to an on-premises data center with cloud based storage

Three types of storage interfaces are there

1) File Gateway

2) Tape Gateway

3) Volume gateway

File Gateway

1) Enables you to sore and retrive files from S3 using file protocol

2) You cannot use S3 transfer acceleration with file gateway

3) You cannot have two gateway's writing to same storage

4) You can have multiple NFS clients accessing single File Gateway

5) NFS Client can read, delete, rename and truncate

6) You can have only one writer in your S3 bucket

7) You can have multiple readers but only one writer

8) You must poll your objects to make sure they are uploaded completely

9) Limit - 10 File shares per gateway and 1 File share per bucket

Tape Gateway

1) Provides backup with ISCSI. Stored in S3 and can be archived to Glacier.

2) Available Sizes100,200,400, 800 GB , 1.5 & 2.5 TB

3) Can have upto 1500 Virtual tapes. Max size - 1 Pb Capacity

4) Available Sizes100,200,400, 800 GB , 1.5 & 2.5 TB

5) Tape Gateway are accessible only through Tape gateway API's

6) They are uploaded completely use IAM role for bucket , cannot use S3 transfer acceleration

Volume gateway

In Volume gateway we have below three configurations. Types of configurations

a) Gateway cached volume

b) Gateway stored volume

c) Gateway Virtual Tape library

Each volume gateway can support 32 volumes

No limit for number of snapshots

Snapshots only accessible from AWS storage gateway cannot accessed using S3 API

Gateway Cached 

Each volume size is 16TB (Max 512 TB)

Cached data is stored in S3 and only frequently accessed data are saved in local

Gateway stored

Each volume size is 32TB (Max 1PB)

Data is stored locally and backed-up  asyncornously by EBS snapshot and stored in S3. 

Provide access to entire data set on low latency

Monitoring Using Cloudwatch following metrics can be monitored – Cache per directory used, cache free, cloud bytes uploaded & downloaded

Cloudformation

Cloudformation 

Gives you the ablity to create a complete AWS environment which consists of different services with the help of a template. We can go head and describe complete environment into the json template and use this template to create the resource when we want for this, using this template we can create the resources whenever we need we can use it.

Concepts of Cloudformation

1) Template - Collection of AWS resources with its attributes.
When we feed this template to cloud formation engine AWS created all the resources in the template which is known as Stack.

To delete all the resources we can simply delte the stack to delete all resources.

2) Stack - Resources created by cloudformation engine with the help of template.

3) Change sets

How to Write a template

We ca

If we want to creat a template from an already created environment then we can use cloudformer. it will ask username nad passoerd (anything) it will create one ec2 instance and cloudformeer installed on the ec2. you can use it via browser it will point to one of region which we choose and all the resources from the region will be created in the template.

About Cloudformation

1) We can use any JSON editor to create template
2) Cloudformation is a IASS (Infratructe as Service)
3) With the help of cloudformation we can quickly replicate the infrastructure
4) Ref: in cloudformation template is we are referring to that resource to create.
5) While creating stack we can either upload template either from Desktop or from S3. 
6) Rollback on failure is b deafult set to false.
7) Stack Policy -  Once you create a stack we can go ahead and update stack. Some time we don't want to update some resources in that case we can use stack policy.
8) Chnage Set - Will Display what are the changes will happen to your stack. We can check this and update the stack.

Dynamo DB

1) Dynamodb is a Fully managed NoSQL. Fast predictable with semaless scalablity

2) All the Data's are stores in SSD's and replicated across multiple AZ's providing built in HA and durablity

3) Components – Tables, items and Attribute

4) Each item in the table has a unique ID or primary key

5) Supports two types of primary key
           (i)  Simple (Partition key)

           (ii) Complex (Partition Key & Sort Key)

6) Supports two types of Index

           (i)  Local secondary index

           (ii) Global secondary index

7) Every index belong to a table called base table

8) Types of read

            (i)  Strong Consistent reads

            (ii) Eventual consistent reads ( Default )

Read and Write capacity units calculation

Read Capacity Units

If your items are smaller than 4KB in size, each read capacity unit will yield one strongly consistent read per second or two eventual consistent reads per second.

Eg 1: Suppose that you want to read 80 items per second from a table and the items are in 3KB size. Then let us see how to calculate the read capacity units. 

Divide the item size of the operation by 4 KB, and then round up to the nearest whole number, as shown below.

3KB/4KB = 0.75  --> 1

So For 80 items 80 X 1 - 80  (So, 80 Units for strong consistent read and 40 for eventual consistent read)

Eg 2:  Calculate the capacity units for a item of  size 10KB and  and for 5 Items.

Divide the item size by 4
10/4(KB) = 2.5 --> 3 

Multiply the Capacity unit with the number of items to get the capacity units for total number of items

3 X 5 -  15 (15 for Strong consistent read and 8 for Eventual consistent read)

Write Capacity Units

In the previous topic we have seen how to calculate read capacity unit. Now will see how to calculate write capacity unit. If your items are smaller than 1KB in size, each write capacity unit will yield one write Per/Sec

Eg 1 : If you want to read 100 items of size 512 bytes. then below is the calculation.

512 Bytes/1 KB = 0.5 --> 1

1 X 100 - 100 Write capacity units

Eg 2 : Let us take second case, where item size is 1.5 Kb and number of items is 10. Then below is the calculation

1.5/1(KB) =1.5 --> 2

For the final number of capacity units, multiply with the value for 1. 2 X 10 - 20 Write capacity units

Limits

1) 5 Global and 5 Local index per table
2) Attribute limit – 64 kb

3) Batch item – 16 MB

4) Total size of item -400 KB

5) Partition key length – 1 byte to 2048 bytes

6) Throughput – Per table      – US East – 40,000 other region's - 10,000

                        – Per Account – US East – 80,000 other region's -  20,000

Supported Data – Number,  String, Binary & Boolean

Note: Empty string not allowed as a valid attribute

Route 53

What is Route 53

Route 53 is a Highly available a Scalable DNS web service 

Functions of Route 53

a) Domain registration
b) DNS service (Can act as an authoritative Name server)
c) Health Checking 
d) Load Balancer

Load Balancer routing policies

a) Simple

b) Latency

c) Weighted

d) Failover

e) Geolocation ( to route based on location)

Simple routing - Use simple routing policy when you have a single resource that performs a give function for your domain. Eg. The IP address is an A record.

Weighted routing - Lets you associate multiple resources with a single DNS name. Weighted is calculated by (Weight for a given resource record set) / (Sum of the weights for the resource record set)

eg: You create three resource record sets, three A records have weight of 1,1 and 3 (sum is 5). On Average route 53 selects each of the first two resource record set one fifth of the time and the third resource  record set three fifth of the time

Latency Based - If application is hosted on multiple regions.You can reduce latency for your users by serving their requests from the region for which network latency is lowest.

Geolocation  - Traffic based on geographic location. Also can be used to restict distribution of content only to spectific location (eg by continent, Country, or by state in US). We cannot create two geo location resource that specify the same geographic location.

Active Passive routing - Primary and secondary should have the same resource record sets. When no health on secondary is set and even primary is un-healthy. Then Amazon will forward the queries to secondary only

General Information

1) Supported Records – A, CNAME, MX, NAPTR, NS, PTR, SOA SOF, SRV, TXT, AAAA 

2) Supports wildcards for all records

3) Does not have a default TTL for any records

4) You can use alias record to map sub domain

5) Can associate multiple IP address with A single record

6) Does not charge for alias queries to ELB (or) other resources

7) Does not support DNS SEC

Limit

Domains  - 50

Hosted Zones - 500

Monitoring

Cloud watch metrics for Route 53 Is free
Health check observation interval – 30 seconds,

Failed  - Required 3 consecutive observation

Traffic Policy 

To route to end user

To use trafice policy – we need to create policy record

Traffic policy – No Charge, Policy record - Chargeable                     

Autoscaling

About Autoscaling

1) Helps you to handle that you have correct number of amazon EC2 instances.

2) Desired Capacity – Ensures group has that many instances

3) Helps to maintain number of instances

4) Can mention minimum and maximum size to maintain

Components

1) Auto Scaling group

2) Launch Configuration – Template used by auto scaling group.

3) Scaling policy

Auto Scaling group 

Contains a collection of EC2 instances that share similar characteristics and are treated as a logical grouping for the purposes of instance scaling and management.

When you create a Auto Scaling group you must specify a launch configuration. You can specify your launch configuration with multiple auto scaling group’s

Example - If an application operates across multiple instances, you might want to increase the number of instances in that group to improve the performance of the application, or decrease the number of instances to reduce costs when demand is low. You can use the 

Auto Scaling group to scale the number of instances automatically based on criteria that you specify, or maintain a fixed number of instances even if a instance becomes unhealthy. This automatic scaling and maintaining the number of instances in an Auto Scaling group is the core functionality of the Auto Scaling service.

An Auto Scaling group starts by launching enough EC2 instances to meet its desired capacity.

The Auto Scaling group maintains this number of instances by performing periodic health checks on the instances in the group. If an instance becomes unhealthy, the group terminates the unhealthy instance and launches another instance to replace it

Imp : An auto scaling group can have only one Launch configuration

We can create an Auto scaling group directly from a EC2 instance. When you use this feature Auto scaling automatically creates a Launch configuration for you as well.

Launch Configuration – Template that an auto scaling group uses to launch EC2 instances. Information specified AMI, Instance Key Pair, Security Groups and Block device mapping

Launch configuration can’t be modified after creation. To change we need to create new launch configuration and update your auto scaling group.

New Launch configuration will be used by new instances, existing instances wont be affected

Creation Launch configuration - Can be created by any one of below methods

1) Using an EC2 instance

2) From an instance and override the block devices

3) Create a Launch configuration and override the instance type 

Auto Scaling Cooldown

Auto scaling cool down period is a configurable setting for your ASG that helps to ensure that auto scaling doesn’t launch or terminate additional instances before the previous scaling activity takes effect.

When you manually scale  your auto scaling period the default is not to wait for the cooldown period. But you can can override the default  and honor the period

If an instance becomes unhealthy, auto scaling does not wait for the cool down period to complete before replacing the healthy instance.

Scaling Plans 

Auto Scaling provides several ways to scale your Auto Scaling group. Below are some of the scaling plans 

Maintain current instance levels at all times 

Manual scaling Scale based on a schedule cale based on demand

Multiple Scaling Policies - An Auto Scaling group can have more than one scaling policy attached to it any given time. In fact, we recommend that each Auto Scaling group has at least two policies: one to scale your architecture out and another to scale your architecture in.

Accessing Methods – CLI, Power shell

Benefits - Better fault tolerance,  Availablity and Cost Management.

Life Cycle – Life Cycle starts when the Auto scaling group launches and instance and puts it into service. Lifecycle ends terminates when you terminate the instance

Charging - No Charges for Auto Scaling, charges are only for resources

Note : You are billed for instances as soon as they are launched including the time that they are not in service

Termination policy

1) Check is Instances are in Multi AZ. If so, it selects the AZ with most instance. If more than one AZ with this configuratiion then it selects the instances that use the oldest launch configuration

2) If multiple instances uses old launch configuration. Choose the one which is near to next billing hour.

3) If more instances are close to next biling hour. then it selects one at random.

Customizing the termination policy 

For customized termination policy, auto scaling first checks the AZ for any imbalance. If an AZ has more instances than the other AZ used by the group. Then Auto Scaling applies your termination policy on the instances of the imbalanced AZ. If the zones are balanced then Auto Scaling applies the termination policy you specified.

Currently supports following custom

              Oldest instance
              Newest instance
              Oldest launch configuration
              Close to Next instance Hour
              Default

Scale out - Fully Configured and EC2 health check passed instance is attached to auto scaling and it enters In Service state. It is counted against the desired capacity

Instance In Service - Instance remains in service until one of the following occurs
             Scale in
             You bu in Standby state
             You detach the instance
             Instance fails health Check

Scale in - Uses termination policy to terminate. It is important to create scale in for each scale out

Attaching already running instance to auto Scaling group - Allows one or more instance to attach to your existing auto scaling group. But must meet the following criteria 

           1) Instance in running state

           2) AMI used to launch must still exist

           3) Instance not member of other Auto Scaling group

          4) If auto scaling has load balancer, instance LB must be in the classic or same VPC. 

If the auto scaling has an attached target group, the instance and the application LB must be in same VPC

Life Cycle Hook - Enables you to perform custom action when auto scaling launches or terminates instances.  Eg: - You Could install or Configure software on newly launched instances or download log files from instance before it terminates. Auto Scaling puts the Instance into a wait state (Pending:Wait or Terminating:Wait). The instance remains in this state until either you tell Auto Scaling to continue or the timeout period ends.

SNS

Fast , Reliable , can send  individual or large receive messages

SNS - Simple notification service. Notification service allows us to send notifications

Publish subscribe paradigm  - We need add the Users or service as subscriptions to a topic. And then if it is Email the Subscriber will get a notification and he need to confirm it, to subscribe to the topic, then only the Subscriber will get a message.

Topic name is unique when created ARN ( amazon resource name ) will be assigned

We can Add multiple type of subscriptions to a topic. For eg: An Email and a notification to a SQS for a single topic.

TTL – default 4 weeks

Delivery status feature – Today we released a new feature for Amazon SNS that enables developers to collect feedback on successful and unsuccessful delivery attempts of their messages to mobile push notification platforms (Apple (APNS), Google (GCM), Amazon (ADM), Windows (WNS and MPNS) and Baidu)

Published message can’t be deleted

Publish Message Option

Publishers send messages to topics. Once a new message is published, Amazon SNS attempts to deliver that message to every endpoint that is subscribed to the topic. In this section you publish a message to the email address you defined in the previous task.

Supported & Un Supported Protocol's

Supported protocol – HTTP, HTTPS, Email, JSON, SQS, SMS, VOIP.

Un Supported protocol  - Blackberry device and MMS and Two Way SMS are not supported by SNS

Billing

Each 64 chunk is billed as 1 request so 256 KB is 4 request

Known Errors 

1) Unable to validate the following destination configurations. Permissions on the destination topic do not allow S3 to publish notifications from this bucket 

Ans: Means the bucket does not have permission to the topic, we need to edit the topic policy and add permission to the respective group in the conditions field