Elastic Cache

What is Elastic Cache

Elastic Cache is a web service that helps in improving web application performance. Allowing you to retrieve the data from fast managed-in cache instead of relying on DB every time.

Advantages of Elastic Cache

a) Reduces load on DB's
b) speed up application response

c) Using this we can save money by offloading some DB activities.

Features of Elastic Cache

a)  Elastic cache has Automatic failure deduction and recovery

b) No minimum fee and no upfront cost  is charged and charges are based on demand usage

c) Free tier is available for testing with a limit of 750 micro node hours for a year.

d) Can be launched in multi AZ

Available Elastic cache engines

As of now Two types of open source engine's are available 
a) Memcached
b) Redis

Redis - adavnced tracking, counting session management and tracking.

memached - simple caching

How to Connect

Note the endpoint of cache cluster in Elastic cache home page.

Can connect from EC2 running on same VPC. Default port for redis 6379

To ineract use telent

telnet <cache name> port name

then can run redis commands

Elastic Beanstalk

Beanstalk is a Platform As A Service solution

With Elastic bean stalk you can quickly deploy and manage application in the cloud.

You simply upload the application, elastic beanstalk automatically handles deployment, capacity provisioning, auto scaling, load balancing and application health monitoring

By default your application is available publicly at myapp.elasticbeanstalk.com for anyone to access

You can move multiple versions of your applications running at same time

You can create upto 75 applications and 1000  application versions by default upto 200 environments across all of your application

You can use elasticbeanstlak  to deploy app in multiple available zones. 

It uses worker environment tier.

With beanstalk you can choose loggin access, OS

Beanstalk runs on amazon linux and window 2012 r2 ami

It takes few minutes to create to run your application

No additional charges for amazon elastic beanstalk you pay only for AWS resources. Actually used to store & run your application. Very easy to deploy With 

What is the Difference between SES and SNS

To send emails through SNS, the email address must be subscribed to an SNS topic. Subscribing an email address to a topic requires opt-in confirmation from the owner of the email address. SNS is orientated toward broadcast (This will create a new list of broadcast recipients. When you send a message to the Broadcast List, it will send to all recipients in the list. The recipients will receive the message as a normal message. When they reply, it will appear as a normal message in your Chats screen; their reply will not be sent to other recipients in the Broadcast List.) Applications (known as publishers) can generate broadcasts to simply send events not knowing who, if anyone, will get them. Receivers (known as subscribers) that want the information subscribe to specific messages via filters. If the message matches a filter, the subscriber is activated (if it’s not already running) and notified of the messages. sending, while SES can support point-to-point sending and has no support for broadcast today. More information about SNS subscription is available in the SNS API reference. By comparison, SES can send to any email address with no prior confirmation.

SES provides full support for a rich email experience, such as HTML, links, and embedded images. SNS allows you to publish notifications across multiple protocols, such as HTTP, SQS, and email. Because SNS is opt-in only, it is easier for us to ensure good email deliverability with SNS, and we can pass this cost saving onto our customers: SNS has a lower price per email than SES.

Points to be considered to migrate an application from On Premises to AWS

Below are some of the tips how to Design an application migration and which component has to placed in which level below are some of the tips for the same

1) If it is publicly accessed web application then we need to design in such a way that initial request is hitting Route 53 then to ELB

Request will hit Route53 from there request will be forwarded to ELB

2) Number of Subnet's to be created

a) Management subnet, ( Antivirus, to check valid Request), 

In Management subnet we will have all management related activities like antivirus, firewall... Even we can place ELB will be in management subnet.

b) Web Subnet

Where web application resides. This subnet can either be public or priavte based on application. Inbound rule for this subnet will be like  all traffic from management

c) Application Subnet 

Where the actual application resides. Inbound for this subnet will be like all traffic from web subnet  and outbound will be like all traffic to web and data db subnet

d) DB subnet 

Ports for application for DB will be different, so it is better to create a seperate subnet for DB. Because opening the ports for application in Security group will open the port to DB instances as well. So it is better to create different subnet. Inbound - only from application

Available tools to migrate.

racemi - migration 80 mb chunks, during migration it will create a timestamp replica for disk

migration

Datacenter to Cloud migration TIPS

Points to be noted to migrate a Server to cloud

1) On Server front

We need to make a note of below details before creating an instance in cloud.

a) CPU

b) Memory

c) Disk

d) NFS/CIFS (shared) volume

e) List of process

f)  Process wise memory

g) port details (iptables)

h) environment

i)  Cluster

2) Application Details

technology stack

internal/external users

how user authentication

cluster

Is technlogy stack upgradable

3) Network Details

Load balancer,

connection between any management subnet

generic n/w arch

4) Managment Details

what antivirus (for license purpose)

AD DNS details

Firewall details

CLI Command's List

For CLI

we need python-pip

pip install awscli

We need to use configure by using aws-configure command

aws configure - to configure access key in ec2

S3

aws s3 ls s3://mybucket

aws s3 cp myfolder s3://mybucket/folder --recursive

aws s3 sync myfolder s3://mybucket/folder --exclude *.tmp

aws s3 mb s3://mybucket

aws s3 rb s3://bucket-name

aws s3 rb s3://bucket-name --force

The sync command has the following form. Possible source-target combinations are:

Local file system to Amazon S3

Amazon S3 to local file system

Amazon S3 to Amazon S3

However, you may supply the --delete option to remove files or objects from the target not present in the source.

EC2

aws ec2 describe-instances - Each instance detail with start with keyword reservations

aws ec2 describe-vpcs

aws ec2 describe-subnets

aws ec2 run-instances

aws ec2 describe-instances

aws ec2 start-instances --instance-ids i-12345678c

aws ec2 create-instances

1)  To create snapshot from command line

aws ec2 create-snapshot --region $region --description $ec2_snapshot_description --volume-id $ebs_selected --output text --query SnapshotId 2>&1

volume id - is the id for the respective ebs volume

2) Create New IAM role

$aws iam create-instance-profile --instance-profile-name YourNewRole-Instance-Profile

$aws iam add-role-to-instance-profile --role-name YourNewRole --instance-profile-name YourNewRole-Instance-Profile

3) Attach Role to existing instance

$aws ec2 associate-iam-instance-profile --instance-id YourInstanceId --iam-instance-profile Name=YourNewRole-Instance-Profile

$aws ec2 describe-iam-instance-profile-associations

4) Replace existing IAM role

$aws ec2 replace-iam-instance-profile-association --association-id YourCurrentAssociation-id --iam-instance-profile Name=YourReplacementRole-Instance-Profile 

AWS Disaster Recovery

AWS Disaster recovery methods

a) Backup and Restore

b) Pilot light

c) Warm Standby

d) Multisite

Backup and Restore

Data is backed up to tape and sent to off site regularly

Pilot light

Minimal version of the service is always running in the cloud.

Quicker recovery than backup and restore method

Warm Standby

Scaled down version of a fully functional environment

Multisite

Active Active Configuration

Linux AMI Virtualization types

Linux Amazon Machine Images use one of two types of virtualization:

a) Paravirtual (PV)

b) Hardware virtual machine (HVM).

The main difference between PV and HVM AMIs is the way in which they boot and whether they can take advantage of special hardware extensions (CPU, network, and storage) for better performance.

For the best performance, we recommend that you use current generation instance types and HVM AMIs when you launch your instances.

HVM AMIs

HVM AMIs are presented with a fully virtualized set of hardware and boot by executing the master boot record of the root block device of your image. This virtualization type provides the ability to run an operating system directly on top of a virtual machine without any modification, as if it were run on the bare-metal hardware. 

HVM guests can take advantage of hardware extensions that provide fast access to the underlying hardware on the host system. HVM AMIs can take advantage of enhanced networking and GPU processing. (In order to pass through instructions to specialized network and GPU devices, the OS needs to be able to have access to the native hardware platform; HVM virtualization provides this access)

All current generation instance types support HVM AMIs. The CC2, CR1, HI1, and HS1 previous generation instance types support HVM AMIs.

To find an HVM AMI, verify that the virtualization type of the AMI is set to HVM, using the console or the describe-images command.

PV AMIs

PV AMIs boot with a special boot loader called PV-GRUB, which starts the boot cycle and then chain loads the kernel specified in the menu.lst (menu.lst -> ./grub.conf) file on your image. Paravirtual hosts cannot take advantage of special hardware extensions such as enhanced networking or GPU processing. 

The C3 and M3 current generation instance types support PV AMIs. The C1, HI1, HS1, M1, M2, and T1 previous generation instance types support PV AMIs.

To find a PV AMI, verify that the virtualization type of the AMI is set to paravirtual, using the console or the describe-images command.

PV on HVM

Paravirtual guests traditionally performed better with storage and network operations than HVM guests because they could leverage special drivers for I/O that avoided the overhead of emulating network and disk hardware, whereas HVM guests had to translate these instructions to emulated hardware. 

Now these PV drivers are available for HVM guests, so operating systems that cannot be ported to run in a paravirtualized environment (such as Windows) can still see performance advantages in storage and network I/O by using them. With these PV on HVM drivers, HVM guests can get the same, or better, performance than paravirtual guests.

AMI Launch Permissions

Public - Launch permission to all

Explicit - Launch permission to special AWS accounts

Implicit - Only owner can launch

How are you charged AMI Launch Permissions

With Instance store AMI every time you customize  and create new one all the parts are stored in Amazon S3 for each AMI. So storage footprint for each customized AMI is full size of AMI.

For EBS backed AMI each time you customize and create new one only the changes are stored  for subsequent AMI. resulting in low storage.

We can convert instance store AMI to EBS backed AMI but lot of steps need to be followed.

Copying of AMI is not chargeable.

launch pemrission, user defined tags or s3 permission or not copied from source AMI  o new AMI.

Making AMI public will take some time and available only to that region.

You can create boot strap script to  customize your AMI  to install software on startup

Difference Between NACL and Security Group

Network Access control lists are applicable at the subnet level, so any instance in the subnet with an associated NACL will follow rules of NACL. That's not the case with security groups, security groups has to be assigned explicitly to the instance.

By default your default vpc, will have a default Network Access Control List which would allow all traffic , both inbound and outbound. If you want to restrict access at subnet level it's a good practice to create a custom NACL and edit it with the rules of your choice and while editing you could also edit subnet associations , where you associate a subnet with this custom NACL, now any instance in this subnet will start following NACL rules

NACLs are stateless unlike security groups. Security groups are statefull ,if you add an inbound rule say for port 80, it is automatically allowed out, meaning outbound rule for that particular port need not be explicitly added. But in NACLs you need to provide explicit inbound and outbound rules

In security groups you cannot deny traffic from a particular instance, by default everything is denied. You can set rules only to allow. Where as in NACLs you can set rules both to deny and allow. There by denying and allowing only the instances of your choice. You get to take more granular decisions.

Security groups evaluate all the rules in them before allowing a traffic . NACLs do it in the number order, from top to bottom like, if your rule #0 says allow all HTTP traffic and your rule #100 says don't allow HTTP traffic from ip address 10.0.2.156 , it's not gonna work, because rule #0 has already allowed traffic. So it's good practice to have your deny rules first in NACL and followed by allow rules. AWS best practice is to number your rules in increment of 100s in NACL.

Accessing internet from Private Subnet

To Access internet from Private subnet. We have two methods

a)            By using NAT Gateway

b)          By Using NAT Instance

By Using NAT Instance

1) Launch the NAT Instance using ( AMI to be used : Amazon NAT AMI ) in public Subnet

2) Launch the instance in private subnet

3) Assign EIP to the NAT Instance

4) Disable Source/Destination Check for the instance (Select Instance -> Actions -> Networking -> Uncheck Source/Destination check )

5) Update the IPTables of NAT table like below (i.e) POSTROUTING to MASQUERADE

# iptables -t nat -A POSTROUTING -o eth0 -s <privae subnet>  -j MASQUERADE

Modify the Security Group of NAT instance inbound rule to all or the port from PVT subnet

Modify the Route table of the PVT subnet for 0.0.0.0/0 to point to NAT Instance

cat <<EOF | sudo tee /etc/sysctl.d/custom_nat_tuning.conf

# for large instance types, allow keeping track of more

# connections (requires enough RAM)

net.ipv4.netfilter.ip_conntrack_max=262144

EOF

sudo sysctl -p /etc/sysctl.d/custom_nat_tuning.con

/usr/local/sbin/configure-pat.sh

By Using NAT Gateway

1) Launch the NAT Gateway in public Subnet. 

2) Launch the instance in private subnet

3) Assign EIP to the NAT Gateway

4) Modify the Route table of the PVT subnet for 0.0.0.0/0 to point to NAT Gateway

Displayed error	Reason	Remedial steps
Subnet has insufficient free addresses to create this NAT gateway	The subnet you specified does not have any free private IP addresses. The NAT gateway requires a network interface with a private IP address allocated from the subnet's range.	You can check how many IP addresses are available in your subnet by going to the Subnets page in the Amazon VPC console, and viewing the Available IPs field in the details pane for your subnet. To create free IP addresses in your subnet, you can delete unused network interfaces, or terminate instances that you do not require.
Network vpc-xxxxxxxx has no Internet gateway attached	A NAT gateway must be created in a VPC with an Internet gateway.	Create and attach an Internet gateway to your VPC. For more information, see Attaching an Internet Gateway.
Elastic IP address eipalloc-xxxxxxxxcould not be associated with this NAT gateway	The Elastic IP address that you specified does not exist or could not be found.	Check the allocation ID of the Elastic IP address to ensure that you entered it correctly. Ensure that you have specified an Elastic IP address that's in the same region in which you're creating the NAT gateway.
Elastic IP address eipalloc-xxxxxxxxis already associated	The Elastic IP address that you specified is already associated with another resource, and cannot be associated with the NAT gateway.	You can check which resource is associated with the Elastic IP address by going to the Elastic IPs page in the Amazon VPC console, and viewing the values specified for the instance ID or network interface ID. If you do not require the Elastic IP address for that resource, you can disassociate it. Alternatively, allocate a new Elastic IP address to your account. For more information, see Working with Elastic IP Addresses.
Network interface eni-xxxxxxxx, created and used internally by this NAT gateway is in an invalid state. Please try again.	There was a problem creating or using the network interface for the NAT gateway.	You cannot fix this error. Try creating a NAT gateway again.

Comparison of NAT Instances and NAT Gateways

The following is a high-level summary of the differences between NAT instances and NAT gateways.

Attribute	NAT gateway	NAT instance
Availability	Highly available. NAT gateways in each Availability Zone are implemented with redundancy. Create a NAT gateway in each Availability Zone to ensure zone-independent architecture.	Use a script to manage failover between instances.
Bandwidth	Supports bursts of up to 10Gbps.	Depends on the bandwidth of the instance type.
Maintenance	Managed by AWS.You do not need to perform any maintenance.	Managed by you, for example, by installing software updates or operating system patches on the instance.
Performance	Software is optimized for handling NAT traffic.	A generic Amazon Linux AMI that's configured to perform NAT.
Cost	Charged depending on the number of NAT gateways you use, duration of usage, and amount of data that you send through the NAT gateways.	Charged depending on the number of NAT instances that you use, duration of usage, and instance type and size.
Type and size	Uniform offering; you don’t need to decide on the type or size.	Choose a suitable instance type and size, according to your predicted workload.
Public IP addresses	Choose the Elastic IP address to associate with a NAT gateway at creation.	Use an Elastic IP address or a public IP address with a NAT instance. You can change the public IP address at any time by associating a new Elastic IP address with the instance.
Private IP addresses	Automatically selected from the subnet's IP address range when you create the gateway.	Assign a specific private IP address from the subnet's IP address range when you launch the instance.
Security groups	Cannot be associated with a NAT gateway. You can associate security groups with your resources behind the NAT gateway to control inbound and outbound traffic.	Associate with your NAT instance and the resources behind your NAT instance to control inbound and outbound traffic.
Network ACLs	Use a network ACL to control the traffic to and from the subnet in which your NAT gateway resides.	Use a network ACL to control the traffic to and from the subnet in which your NAT instance resides.
Flow logs	Use flow logs to capture the traffic.	Use flow logs to capture the traffic.
Port forwarding	Not supported.	Manually customize the configuration to support port forwarding.
Bastion servers	Not supported.	Use as a bastion server.
Traffic metrics	Not supported.	View CloudWatch metrics.
Timeout behavior	When a connection times out, a NAT gateway returns an RST packet to any resources behind the NAT gateway that attempt to continue the connection (it does not send a FIN packet).	When a connection times out, a NAT instance sends a FIN packet to resources behind the NAT instance to close the connection.
IP fragmentation	Supports forwarding of IP fragmented packets for the UDP protocol. Does not support fragmentation for the TCP and ICMP protocols. Fragmented packets for these protocols will get dropped.	Supports reassembly of IP fragmented packets for the UDP, TCP, and ICMP protocols.

Scenario Based Questions

1. How to migrate an EC2 from one region to anohter

a) take snapshot of the ec2 by right click and create image option

b) copy the AMI to different region

c) then from the new AMI create an EC2

2) How to change the users secret access key

Secret access key cannot be changed. New key need to be generate and assign the user with policy IAMUserSSHKeys. So that next user can manage his own ssh keys.

3) Allow IAM users to change their own passwords

In IAM account Settings, enable Allow users to change their own password, and then click Apply Password Policy.

4) I accidentally deleted my default VPC in AWS. What to do now?

You can open a support ticket and they will re create it for you.

5) In S3

If there is no Edit in the file in S3 and if you upload another file of same name, it is overwritten if versioning is not enabled. A new version is created if version is maintained.

b) Within the Region data is replicated into multiple availablity zone's.

c) Ownership of the object is not transferable.

d) Version ID is the version id for the object and in combination with the key helps to unique

e) Subresources helps provide additional information for an object

Elastic Load Balancing

ELB - Elastic Load Balancer

It is a managed service by AWS (Scalability and Availablity is managed by AWS). Mainly it used for distributing traffic between instances. Once if the instance is registered with ELB, it performs continuous Health check. Once if it finds the unhealthy instance ELB will stop sending the traffic.

ELB can either be public or private it is based on the subnet where we are launching it. It is highly recommended to use 2 AZ's for high availablity. 

ELB creates instances in the subnet's in backend which we cannot see. Private IP's for ELB will be assigned from the subnet which we have selected for the ELB. ELB will expand based on the traffic. Traffic will be distributed to Availablity zone's in round robin manner.

Note: ELB can be accessed only by DNS CNAME and not by IP.

Cross zone load balancing 

It will be useful if we are using multiple zones and if the number of instances in the zones are not equal. For example, If we have 6 Instances in 2 AZ's and in Zone A 4 instance and Zone B 2 instance. ELB will blindly forward the requests in Round robin manner to Zone A & B which causes the instances in Zone B to handle more load compared to the instances in Zone A. To Over com this we can enable cross zone load balancing option.

If cross Load Balancing is enabled then ELB will take the count of all the instances in all the availablity zone's  and will forward the traffic to instance which has least number of connections.

Pre - Warming

If you expect a sudden increase in your hits. For eg: from 1 to 10,000 hits in particular time then it will take some time for ELB to auto scale in backend. To avoid this get your ELB pre warmed by contacting AWS support. They will configure the load balancer to have the appropriate level of capacity based on the traffic that you expect.  

They also need some more details like Start and End dates, Expected flash traffic, Expected request rate per second and the total size of the typical request/response.

Connection Draining 

If enabled, time you give to the instances to complete the already establieshed connection for the unhealthy instances. But it wont send any new traffic to those unhelathy instance.

Internal load balancer - Load balancer launched in the private network.

If an instance is using ELB that doesnt mean the EC2 can download files from internet.