Wednesday, August 23, 2017

All About IAM (Identity and Access Management)

List of Topics

1) What is IAM
2) Features of IAM
3) Groups, Roles and Types of Policy
4) Difference Between Managed Policy and Inline Policy
5) Access Key
6) Federating Existing users

1) What is IAM

IAM - Identity and access Management - We can manage different groups, users  their policies and roles. Every account has a root account which we would have created during initial account creation in AWS. Using root account for day to day operations is not recommended as it has unrestricted access for all resources on AWS. AWS is free of cost service which includes AWS security token service also (STS).

2) Features of IAM

a) There is no charge for using IAM
b) Following ways can be used to interact with AWS - Console, CLI (Programatic access )
c) If we add a user to group, user will inherit the permissions of the group.
d) Username's cannot be changed in Management console. To change the username we need to use either CLI or API.
e) Deleting an user will delete Users group, password and access keys.
g) In case of a conflict If there are two policies, one is allow permission to a bucket and other is deny access to a bucket. Then deny will apply. In simple only if there is any undisputed access then only allow policy will work.
h) By Default IAM is highly available by replicating data to multiple servers.

3) Groups, Roles and Types of Policy

Group

Whatever the policy we are attaching to a group. Users in the group also will get attached to the policy. We can have multiple user in one group and One users in multiple group. But group to group attaching is not possible.

Roles

In order to authorize a particular AWS service to use/call API other AWS service. In other words, used to give permission for a particular AWS service to use other AWS service. Role names must be unique within your AWS account.

Roles can be attached to running EC2 instance as well by using CLI. Roles can be attached to an EC2 only while launching it.

If role is assigned to a EC2 then there is no requirement to use Secret access key and -access key. We can direct execute the CLI commands if it is insatalled.

The simplest way to use roles is to grant your IAM users permissions to switch to roles that you create within your own or another AWS account. They can switch roles easily using the IAM console to use permissions that you don't ordinarily want them to have, and then exit the role to surrender those permissions. This can help prevent accidental access to or modification of sensitive resources.

For more complex uses of roles, such as granting access to applications and services, or federated external users, you can call the AssumeRole API. This API call returns a set of temporary credentials that the application can use in subsequent API calls.


Federated users sign in by using credentials from an identity provider (IdP). AWS then provides temporary credentials to the trusted IdP to pass on to the user for including in subsequent AWS resource requests. Those credentials provide the permissions granted to the assigned role.

Policy

Policy is also known as collection of permissions. By Default users can control anything in the account if no policy is assigned. This can be controlled by assigning the specific policy to user. There are many predefined policy available in AWS and even we can create our own custom policy also.

Types of Policy There are two types of IAM policy available

    a) User based
    b) Resource based

Which means a policy will be assigned either to an User or resource.

Some predefined policy.

Power user policy - User has full access without IAM management
Admin policy - User has full access including IAM

5) Difference Between Inline Policy and Managed Policy

Inline Policy

An inline policy is a policy that is specific to a user, group, or role. You can create a policy and embed it in a principal entity, either when you create the principal entity (a user, group, or role) or later.

Managed Policy (Managed either by AWS or by Customer)

Managed policy is a standalone policy that is created and administered by either by AWS/Customer. Standalone policy means that the policy has its own Amazon Resource Name (ARN) that includes the policy name. For example, arn:aws:iam::aws:policy/IAMReadOnlyAccess is an AWS managed policy. You can attach the policies to multiple principal entities (a user, group or role) in your AWS account. When you attach a policy to a principal entity, you give the entity the permissions that are defined in the policy.

4) Access Key

Access key is used to Make API Calls. And access key has the following two parts

1) Secret access and Keys and
2) Access Key ID

Users need their own access keys to make programmatic calls to AWS from the AWS Command Line Interface (AWS CLI), Tools for Windows PowerShell,

When you create an access key, IAM returns the access key ID and secret access key. You should save these in a secure location and give them to the user.

Secret access key is accessible only at the time you create it. If a secret access key is lost, you must delete the access key for the associated user and create a new key.

By default, when you create an access key, its status is Active, which means the user can use the access key for AWS CLI. Each user can have two active access keys, You can give your users permission to list, rotate, and manage their own keys by  assigning the user with policy IAMUserSSHKeys. So that next user can manage his own ssh keys.

5) Identity Federation

A federated identity means linking a person's electronic identity, In AWS federation has following two methods.

a) Users have identities in corporate directory
    (i)  If corporate directory with  SAML (2.0) you can configure SSO
    (ii) If MS AD, you can use AWS DS
b) Users already have internet identities

Using well known web identity like Facebook, Google
Use Amazon Cognito, it acts as an id broker and does much of the federation work for you, if you don't use it. You must write a code that interacts with IDP then calls API

6) Limits

Groups in an AWS account - 100
Users in an AWS account - 5000 (If you need to add a large number of users, consider using temporary security credentials. For more information about temporary security credentials, go toTemporary Security Credentials.)
Roles in an AWS account - 250
Instance profiles in an AWS account - 100
Roles in an instance profile - 1 (This limit of one role per instance cannot be increased.)
Groups a user can be a member of - 10
Access keys assigned to a user - 2
Access keys assigned to the AWS root account - 2

Wednesday, August 9, 2017

What is the use of Alias Record in Route 53

As per DNS restriction we cannot point an CNAME to APEX (postsbylukman.com) record is not possible. 

Apex Record Sample - postsbylukman.com (sometimes called the "root domain" or "naked domain")

And www.postsbylukman.com is not an APEX record

Possible : www.yourdomain.com CNAME some-id.ec2.amazonaws.com

Not Possible : yourdomain.com (awspostsbylukman.com) CNAME some-cnameaws.com

The reason is DNS RFC requires "zone apex"  to be an "A Record," not a CNAME. This means that with most DNS providers you can setup a subdomain CNAME to point to EC2, but you cannot setup your root domain as a CNAME to point to EC2.

To Avoid this only AWS come up with a record a ALIAS record.

S3 and EBS Storage Modes

What is block level storage and What is object level storage

S3 uses object level Storage and EBS uses block level storage. Please find more info about block level storage and about object level storage below.

Block level Storage

In block level storage your HDD will be divided  into small pieces called blocks. For eg:  Let's say you have an file of size 10 MB and you are saving it to your HDD of your PC/Laptop. File will take space on HDD based on the block size of HDD, if the block size is 512 KB then it will take 20 blocks in the HDD. If the file is opened again and if the contents of the file are modified. Then is it required to update all blocks ?. The Answer is NO, it is enough to modify the blocks where change has happened. 

Object Level Storage

In object level storage, lets take the same. If the file is opened again and if the contents of the file are modified. Then is it required to update all blocks ?. The Answer is YES in this case, the complete file need to be flushed out and copied again.

TIP : For fast read and write, block level storage is the right option. For continuous read and rare write then object level storage is better is the better Option

Friday, July 14, 2017

S3 Bucket Level Operations

Bucket Level Operations

Here are the list of some bucket level operations with their explanation.

a) Versioning

Means keeping multiple variants of an object in the same bucket. Once enabled for a bucket. We can't disable it. Also Even after enabling it will take effect only for new objects. This option can be used to protect from unintended overwrites and deletions and allows us the ability to retrieve and restore deleted objects or rollback to previous versions

b) Multipart Delete

To be used to delete large number of objects from S3. The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.

Note : Both empty and Non-empty buckets can be deleted

c) Encryption

(i)  Data In transit – By using SSL
(ii) Data at rest     – Client side encryption - Encrypt and Upload
                            – Server side encryption ( Req AWS to encrypt before saving)
o   with Amazon s3 managed keys (SSE –S3)
o   with KMS managed keys (SSE –KMS)
o   with customer provided keys (SSE – C)

d) Cross Region replication

This is a Bucket level feature which enables automatic asynchornous copying of objects of one bucket to a another bucket in different AWS region.

To activate Add replication configuration to your source bucket and provide info of destination bucket.

We can request Amazon S3 to replicate all (or) a subset of object with specific key name prefix. Replica will have same key name, metadata, creation time, owner, version, ACL and storage class.

Requirements for cross region replication -
           Version must be enabled
           Different region
           Between Different account is also possible (i.e) Cross account

When using VPC with S3, use VPC S3 endpoints as are horizontally scaled, redundant, and highly available VPC components

e) Server access logging

In Order to track requests fir access to your bucket, you can enable this option. Each request has "requestor, bucket name, time, status and error code:

There is no extra charge for enabling server access logging on an Amazon S3 bucket; however, any log files the system delivers to you will accrue the usual charges for storage.

(You can delete the log files at any time.) No data transfer charges will be assessed for log file delivery, but access to the delivered log files is charged the same as any other data transfer.

What is VPC

1) What is VPC

VPC, Virtual private cloud. if you choose tenancy as dedicated then you will be able to launch only Dedicated instances or dedicated hosts

2) Creating VPC

While creating VPC we may see below options and need to select the values.A short information about the options

(i) Tenancy - While creating VPC if you choose tenancy as dedicated then you will be able to launch only Dedicated instances or dedicated hosts

(ii) CIDR block - It represents range of IP Address Eg; 10.0.0.0/26 Start will be 10.0.0.0 and rang will be [ 2 ^ (32 - 26)  = 2^6 = 64 ] and so the end will be 10.0.0.63

(iii) Subnet - It represents range of IP Address. Amazon reserves first 4 and last 1 IP of every subnet. Default subnet within default VPC are assigned /20 netblocks.

3) VPC Resources & Settings

(i) NACL - Network Access controlled List. - Subnet level Traffic filtering .

(ii)  Internet Gateway: VPC side of a connection to the public Internet. 1 Per VPC

(iii) NAT Gateway: A highly available, managed Network Address Translation (NAT) service for your resources in a private subnet to access the Internet.

(a) Modify auto assign IP Setting - If true, enables auto assign public for Instances. Enable auto-assign public IPv4 or IPv6 addresses to automatically request an IP address for instances launched into this subnet.

3) Resoure Location

VPC - Specific to Region. We can't mention AZ.
Subnet - Specific to AZ

4) VPC Settings

Edit DNS resolution -
Edit DNS host name - If checked, instances will get private hostname.

CIDR block -

VPC (Virtual Private Cloud) - Logically isolated virtual network in the AWS cloud. You can define VPC's IP address space from a range you select

VPC wit single public subnet -

Using an Elastic IP address (EIP) enables an instance in a VPC, which is otherwise private, to be reached from the Internet through an Internet gateway (for example, it could act as a web server).

4) VPC options

(i)   VPC wit single public subnet
(ii)  VPC with public/private subnet
(iii) VPC With public/private subnet and Hardware VPN access
(iv) VPC with private subnet only and Hardware VPN access

5) Components of Amazon VPC?

Amazon VPC comprises a variety of objects that will be familiar to customers with existing networks:

Hardware VPN Connection: A hardware-based VPN connection between your Amazon VPC and your datacenter, home network, or co-location facility.

Virtual Private Gateway: The Amazon VPC side of a VPN connection.

Customer Gateway: Your side of a VPN connection.

Router: Routers interconnect subnets and direct traffic between Internet gateways, virtual private gateways, NAT gateways, and subnets.

Peering Connection: A peering connection enables you to route traffic via private IP addresses between two peered VPCs.

VPC Endpoint: Enables Amazon S3 and Amazon DynamoDB access from within your VPC without using an Internet gateway or NAT, and allows you to control the access using VPC endpoint policies.

Egress-only Internet Gateway: A stateful gateway to provide egress only access for IPv6 traffic from the VPC to the Internet

6) Charge – No Charge for VPC & VPC peering

7) VPC Peering – Is between availability zone’s only, Peering must non-overlapping IP range.

8) Limits

VPC           – 5 per region
Subnet       – 200 per VPC
IG               – 1 per VPC
Virtual private gateway – 5 per region
Customer Gateway       – 50 per region

9) Security group

Default security group config – Allow All outbound and No inbound

10) General Information's

1) Default VPC are assigned range of 172.31.0.0/16 in size
2) Currently AWS supports VPC between /28(In CIDR) & /16 in size.
3) We can't change the size of VPC. You must terminate and create new one
4) You can't assign IP address for multiple instances simultaneously
5) We cant dtach eth0

11) VPC Peering 

Connecting of two non CIDR overlapping VPC's. Peering is available between same region. Peering between different AWS account is possible.

Conditions/Restrictions

(i) No Tranisitive Peering (Eg:Let us there are 3 VPC's A,B,C. A is peered between B & C that does'nt mean that VPC C can connect B.
(ii) No Edge routing
(iii) No NAT routing

About Peering

(i)  We need to update route tables of the VPC after Peering. Like for the CIDR block of VPC 2 and entry in VPC 1 route table need to be created with the destination as Peer name.
(ii)  We cannot use the Security group of Other VPC even it is peered
(iii) Windows instances cannot boot correctly if launched into a VPC with ranges from 224.0.0.0 to 255.255.255.255 (Class D and Class E IP address rang


Saturday, July 8, 2017

Difference between Public IP and Elastic IP Address

Public IP and Elastic IP Address

a) Public IP address 


Instance with Public IP address will get an external DNS name (example, ec2-506-0-198-88.compute-1.amazonaws.com)  name. External DNS name will be able to resolve outside the network and to the private IPv4 address of the instance from within the network of the instance. The public IP address is mapped to the primary private IP address through network address translation (NAT).

Public IP address can't be manually associated or disassociated for your instance. AWS releases or assigns the public IP address  in following cases

1) When instance is  stopped or terminated. Stopped instance receives a new public IP
2) When you associate an Elastic IP address with your instance Public IP address for your instance will be released.
3) When you associate an Elastic IP address with the primary network interface (eth0) of your instance in a VPC.

4) When you disassociate the Elastic IP address from your instance, it receives a new public IP address.

Reboot of EC2 WON'T change the public IP address of your instance changes

If you require a persistent public IP address that can be associated to and from instances as use an Elastic IP address.

b) Elastic IP address 

An Elastic IP address is a static IPv4 address designed for dynamic cloud computing. Elastic IP address will be associated with your AWS account.

Elastic IP address doesn’t incur charges as long as the below conditions are met

1)  The instance associated with the Elastic IP address is running.

If above conditions are not met then you will be charged for the Elastic IP Address.

If you’ve stopped or terminated an EC2 instance with an associated Elastic IP address and you don’t need that Elastic IP address any more, consider disassociating or releasing the Elastic IP address.

Note: Once if the Elastic IP address is released, you can’t get the same Elastic IP address again, you have to get a different Elastic IP address.

EC2 Tenancy Models

Here are different types of tenancy model available which we need to select while creating the EC2. Below are the list of models with explanation

a) Shared Tenancy 

The physical Hardware where your EC2 is hosted will be shared with other customers also. Pricing is less as the Hardware is shared with other customer's. When the instance is Stopped and Started the underline physical server will change but  Reboot will not change the underline hardware.

b) Dedicated Instances 

The physical hardware is completely dedicated for you and won't be shared with other customer and only your EC2 instances will be hosted there. In Simple terms, Hardware dedicated for your account. But during start and stop of the server the underline Hardware will change and we may get new dedicated hardware. And your shared instances (same account) wont be launched in the underlying hardware only dedicated instances of particular type will be launched in the same hardware.

For example: Requesting a m4.large dedicated instance could end with your instance on a server exactly the size of an m4.large.

In addition to instance charge of per hour rate an additional of 2$ will be charged per hour per region. This 2$ will be same if it is 10 instance or 1 instance. As it will be charged per hour per region

c) Dedicated Hosts 

Only difference between Dedicated instance and Dedicated host is even after stop & start underlying hardware will not change. This method is useful when we are going for a hardware based license model. Billing will be high as we are using full host.

As we get the complete hosts, irrespective of utilization we will pay the complete charge as the host is dedicated for You. This is bit costlier than others.  

Number of instances per dedicated hosts calculation is available in AWS help page. For example if we select m4 as our dedicated hosts then we can have 22 large m4 or 11 xlarge

Monday, June 19, 2017

Questions and Answers


The AWS Policy Generator is a tool that enables you to create policies that control access to Amazon Web Services (AWS) products and resources. For more information about creating policies, see key concepts in Using AWS Identity and Access Management. Here are sample policies.


1. Does multiples ENI’s increase the availability of the machine.?

ANS: No, if instance fails all the ENIs attached to that instance would not be of any use. Multiple ENIs do not increase Availability or Bandwidth of the machine. Rather, they are there for the isolation purpose.

2. Normally in an ON-PREM INFRA, we used to have muti NIC for the webserver not to have a single point of failure in case of one NIC CARD fails? Where as in AWS its a virtual world and I assume its a managed service right? we don't have to worry about the Primary failing at all ?

ANS: On AWS, we handle that scenario via Auto-scaling Group. So that if a machine becomes unreachable, another instance takes its place.

3. The primary ENI can't be detached, Becoz that's the one which makes the instance to have a public and/or private IP right?

ANS: Not so. It is more of a restriction from AWS implementation perspective. In future, they might start allowing it (as you can detach ROOT EBS volume for a stopped machine).

4. Under health check, the ping target, I gave the path index.html. let's assume I have 2 instances and if I want the index.html to reside in a different location for all the 2 instance, let's assume for the 1st one, it's under var/www/HTML and for the second one if its under var/www ..does the ping target as the intelligence to check the file irrespective if its location?

ANS: Given the fact that all the instances behind an ELB are generally kept identical, the path of the file has to be same. Though, if you have some specific use-case for the above scenario, it could be achieved using Application Load Balancer. But, in Classic ELB, it has to be same path. 

5. So the best use case for multi eni would be, Where in a. In a single large instance with multi ENI, that one instance can be part of both private and public subnet b. Having said that, I can have web server facing the internet in the public ENI and the database also in the same instance in the private ENI and configure the security group accordingly.

ANS: Sorry, Incorrect! When you create ENI, you can see that its scope is a Subnet (same way scope of an EC2 is also Subnet). Hence, an EC2 and an ENI would be there only in one subnet (not two).
You can use multi ENI to give 2 different IPs to 2 different user groups. E.g. On an EC2 open port 8080 on IP1 and port 22 on IP2. Also, attach different security groups in above ENIs. Hence, the users would never know they are accessing the same machine.

6. So with multiNIC, I can have multiple websites on the same server, with each Nic attached to one website?

ANS: You can have multiple websites on one machine even without multi-NIC. Different webservers (like IIS) support the same.

7. you had mentioned that the ELB internally creates instances to manage traffic. Just to expand a little on this, So When this ELB creates the internal instance(i assume its not visible to us ) and it consumes the private IP's too right, Does the internal instance gets created per availability zone or per instance? Coz, I attach an ELB to the instance not to the A.Z ?right? I assume it's something like an agent which gets installed per instance and sort of polling the Domain LB? pls, correct my understanding?

ANS: Yes, those instances won't be visible to users as ELB is a managed service. These instances do use ENIs (and hence Private IPs) from the respective subnets. ELB creates one instance in every subnet (you choose while ELB creation). There is no agent running on your instances (e.g. web server). Also, you register your instances to an ELB and hence tell ELB that it could send traffic to these instances.

8. Under Load balancers / instances, down the page under availability zone, ELB had created 2 AZ's. whereas I had all 3 of my instances in the same A.Z, though I can edit it and remove the other A.Z, why does it create an another AZ reference point when all my instances are in only one AZ? 

ANS: While creating ELB, you can clearly choose which subnets you want it to handle. It is recommended that you give 2 subnets in 2 different AZs (based on principle of HA). ELB will launch an internal instance in the subnets you choose while creating ELB.

9. How to convert On-Demand instance into Reserved instance.

I understand that it's supposed to be simply a billing difference, however, the tool that AWS provides to purchase a reserved instance appears to want to create a new instance. It does not provide an interface to select an existing instance to change billing on.
Super-simple scenario. Consider a customer with a single micro EC2 instance that he wants to turn into a reserved instance. How do I do this?
The "Purchase Reserved Instances" wizard in the console appears to be creating a new instance--a new virtual server. It is asking questions like which platform, which availability zone, etc. I hope you can see that my assumption is that if I go through with an RI purchase, I'll end up with a new micro instance rather than simply telling AWS to bill me differently for my existing instance.
A reserved instance is actually just a reservation and no instance will be created when you purchase one. If you have an existing instance, you simply purchase a reservation matching that instance type, platform, and availability zone. Behind the scenes, the billing system is constantly looking at the running instances and your reservations. When there is a match (same type, platform, az), you automatically get the lower hourly rate.
If you go with a Heavy utilization reservation, you will however be charged for the full number of hours each month no matter if you have an instance or not. With Light and Medium, you are only charged when there is an instance up and running.
A reservation isn't tied to any particular instance, meaning that you can launch and terminate instances as you wish. You still work with EC2 in the exact same way as before.
Thank you. I understand now.
I think for the average dummy like me, the reserved instance purchase process feels too much like setting up a new instance. Easy to feel like I'm doing the wrong thing. Since it is important to select the correct instance type and AZ to get the expected benefit, it would be helpful if their process allowed you to select an existing instance to purchase a reserved instance that matches the instance type and AZ--especially since the one-time fee is non refundable.
Thanks!
Let say I got 3 servers and all of them are in the same AZ, same platform.
can I purchased 2 reserved instance then,
does it means that the 2 servers will be billed by the
reserved intance and the other server will be billed by on demand?
Yes. That's exactly what it means.


Thursday, June 15, 2017

Elastic Cache

What is Elastic Cache


Elastic Cache is a web service that helps in improving web application performance. Allowing you to retrieve the data from fast managed-in cache instead of relying on DB every time.

Advantages of Elastic Cache

a) Reduces load on DB's
b) speed up application response

c) Using this we can save money by offloading some DB activities.

Features of Elastic Cache

a)  Elastic cache has Automatic failure deduction and recovery
b) No minimum fee and no upfront cost  is charged and charges are based on demand usage
c) Free tier is available for testing with a limit of 750 micro node hours for a year.
d) Can be launched in multi AZ

Available Elastic cache engines

As of now Two types of open source engine's are available 
a) Memcached
b) Redis

Redis - adavnced tracking, counting session management and tracking.
memached - simple caching

How to Connect

Note the endpoint of cache cluster in Elastic cache home page.

Can connect from EC2 running on same VPC. Default port for redis 6379
To ineract use telent
telnet <cache name> port name
then can run redis commands

Elastic Beanstalk

Beanstalk is a Platform As A Service solution

With Elastic bean stalk you can quickly deploy and manage application in the cloud.

You simply upload the application, elastic beanstalk automatically handles deployment, capacity provisioning, auto scaling, load balancing and application health monitoring

By default your application is available publicly at myapp.elasticbeanstalk.com for anyone to access

You can move multiple versions of your applications running at same time

You can create upto 75 applications and 1000  application versions by default upto 200 environments across all of your application

You can use elasticbeanstlak  to deploy app in multiple available zones. 
It uses worker environment tier.
With beanstalk you can choose loggin access, OS
Beanstalk runs on amazon linux and window 2012 r2 ami
It takes few minutes to create to run your application

No additional charges for amazon elastic beanstalk you pay only for AWS resources. Actually used to store & run your application. Very easy to deploy With