S3 Bucket Level Operations

Bucket Level Operations

Here are the list of some bucket level operations with their explanation.

a) Versioning

Means keeping multiple variants of an object in the same bucket. Once enabled for a bucket. We can't disable it. Also Even after enabling it will take effect only for new objects. This option can be used to protect from unintended overwrites and deletions and allows us the ability to retrieve and restore deleted objects or rollback to previous versions

b) Multipart Delete

To be used to delete large number of objects from S3. The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.

Note : Both empty and Non-empty buckets can be deleted

c) Encryption

(i)  Data In transit – By using SSL

(ii) Data at rest     – Client side encryption - Encrypt and Upload

                            – Server side encryption ( Req AWS to encrypt before saving)

o   with Amazon s3 managed keys (SSE –S3)

o   with KMS managed keys (SSE –KMS)

o   with customer provided keys (SSE – C)

d) Cross Region replication

This is a Bucket level feature which enables automatic asynchornous copying of objects of one bucket to a another bucket in different AWS region.

To activate Add replication configuration to your source bucket and provide info of destination bucket.

We can request Amazon S3 to replicate all (or) a subset of object with specific key name prefix. Replica will have same key name, metadata, creation time, owner, version, ACL and storage class.

Requirements for cross region replication -

           Version must be enabled

           Different region

           Between Different account is also possible (i.e) Cross account

When using VPC with S3, use VPC S3 endpoints as are horizontally scaled, redundant, and highly available VPC components

e) Server access logging

In Order to track requests fir access to your bucket, you can enable this option. Each request has "requestor, bucket name, time, status and error code:

There is no extra charge for enabling server access logging on an Amazon S3 bucket; however, any log files the system delivers to you will accrue the usual charges for storage.

(You can delete the log files at any time.) No data transfer charges will be assessed for log file delivery, but access to the delivered log files is charged the same as any other data transfer.

What is S3

1) What is S3

Highly-scalable, reliable, and low-latency data storage infrastructure at very low costs.

Can be used to store and retrieve any amount of data, at any time, from within Amazon EC2 or from anywhere on the web

2) Types of S3 Storage and its Durablity/Availablity

S3 has four following storgae Classes, where Glacier is used to archive objects for long term. Different storage class has different pricing.

a) S3 Standard

b) S3 Standard – Infrequent acces

c) Glacier

d) S3 – RRS (Reduced Redundancy Storage)

Durablity -  11 9’s - STD, STD-IA and Glacier &  4 9’s - RRS

Availablity – 4 9’s - STD, RRS  &  3 9's -  STD-IA 

RRS - Storage option that enables customer to reduce their costs for storing non-critical data. Annual expected loss 0.01 %

3) Bucket URL format 

http://<bucketname>.s3.amazonaws.com/<objectpath>

4) About S3 Object

Each S3 object has Data, key and metadata

a) Key  - An unique identifier for an object with bucket. Key is object name. Every object has only one key.

b) Metadata is set of name - value pairs. You can set object metadata the time you upload it. After you upload you cannot modify the object metadata. the only way to modify is make a copy of the object ans set the metadata. the data about the data and is a set of name-value pairs that describe the object for e.g. con tent-type, size, last modified. Custom metadata can also be specified at the time the object is stored.

Version ID is the version id for the object and in combination with the key helps to unique identify an object within a bucket

Subresources helps provide additional information for an object

Access Control Information helps control access to the objects

5) Limits

(i)  Bucket - 100/Account  (If need more request can be raised to AWS)

(ii) File Size - 0 to 5 TB

6) Consistency

a) New objects - Read after Write consistency for all regions

b) Overwrite PUTs and DELETE's - Eventual consistency for overwrite 

7) Requestor Pays

S3 billing happens based on the amount of data stored. And o minimum billing. Amount of data out from the bucket and amount requests received for a object. (Like Reading Object, Listing Object., put get, delete)

If Request pays is enabled the cross acccount owner will pay data out charge and request chrge. But still storage charge will be paid by bucket owner.

Transfer accelartion : let say from banagleotr we want upload some data from blr to tokyo. normally it will go by pubic network. If this enabled searches for one of the nearest edge location (eg blr to chennai public network from there it uses amazon netowrk)

Notes:

a) If there is no Edit in the file in S3 and if you upload another file of same name, it is overwritten if versioning is not enabled. A new version is created if version is maintained.

b) Within the Region data is replicated into multiple availablity zone's.

c) Ownership of the object is not transferable.

d) Version ID is the version id for the object and in combination with the key helps to unique

c) Identify an object within a bucket

e) Subresources helps provide additional information for an object

f) Access Control Information helps control access to the objects