Skip to content

S3

Jump to bottom
abk edited this page Nov 26, 2019 · 1 revision

S3 (Simple Storage Service).

  • S3 is a global service.
  • S3 is NOT a file system or a file system.
  • Replicates objects across multiple AZ across the region
  • Durability is 11-9s
  • Pay as you go.(How many puts and gets).
  • No need to provision a priori.
  • Just store your objects and pay.
  • No upper limit for data storage.
  • 0Bytes (lower limit) to 5TB object size (upper limit).
  • Storage classes (standard, IA, IA-single zone, Glacier)
  • Can use lifecycle policies to save money.
  • Origin for CloudFront CDN
  • Object storage
    • Object is key and value is the data.
  • Great for texts, photos , videos etc. Primary requirement is to store and provide content.
  • Bucket is just container in S3 and doesn’t cost any money.

Bucket facts and fundamentals

  1. Globally unique
  2. Minimum 3 and maximum 63 character - No uppercase or underscores
  3. Must start with lowercase letter or number and no IP addresses formatting
  4. 100 buckets per account and hard 1000 bucket limit upon request
  5. Unlimited objects in buckets
  6. Unlimited total capacity
  7. Key is it’s name
  8. Value is it’s data
  9. Bucket policies are resource policies.
  10. Identity policies can also be used.
  11. ACLs are older style and not recommended by AWS for S3 access.

Storage gateway

  • Piece of s/w that is installed on the datacenter
  • Apps writes to storage volumes and that get written to S3.

PS: There is a charge for data transfer across regions (to and from S3).

Consistency rules

  • Read-After-Write consistency for PUTs of new objects in S3.
  • Eventual consistency for PUTs overwriting existing objects and DELETEs of objects.
  • CAT theory - Consistency, Availability and Tolerance (Partition).

S3 components

  • Buckets (Containers that can hold objects). Consider them like namespaces

    • Can prefix the object so they mimic like folders
    • Can tag the objects.
    • Can enable encryption at bucket level as well.
    • Naming convention
      • Name has to be unique across globally.
      • DNS compliant
      • No uppercase letters or underscores
      • Lowercase letters, numbers and hyphens (Periods for static hosting).
    • Lifecycle rules
      • Automated way to move the data between storage classes.

  • Encryption

    • Server side Encryption
      • SSE-S3 (server side encryption)
Keys are generated by S3 (using KMS on your behalf). Keys are stored with objects in encrypted form.
If you have perms to object you can decrypt the data.
      • SSE-C - Customer provides encryption key
      • SSE-KMS S3 uses master key in KMS
Objs are encrypted using individual keys generated by KMS. Encrypted keys are stored with encrypted obs.
Decryption of obj needs both S3 and KMS key permissions.
    • Client side Encryption
Before object is put in S3, client side encrypts.
      • S3 stores the object with encryption including in transit.

  • S3 features

    • Versioning is at bucket level
    • Storage classes (4)
      • Standard (Most expensive. Best availability, 99.99% availability).
      • Infrequent Access (s3-IA) (99.9% availability, 11-9’s durability).
      • One zone infrequent access (S3 One Zone - IA, 99.5% availability, 11-9’s durability).
      • Glacier - Separate storage service from S3. But can be set as storage class for S3. - Really cheap. (Warm backup).
      • Glacier deep archive. (Cold backup)
      • Intelligent tiering (another storage class)
        • Automated process to move the data between different storage classes.
        • It uses access patterns to move the objects.
          • For eg: if the object is not used for 30 days, it will move to IA
          • If the object is used frequently, the object will be moved to standard s3.
          • You will pay monthly monitoring cost.
          • Obj smaller than 128KB can’t be transitioned into Intelligent Tiering.
          • Reduction of admin overhead of S3 objects.

  • Bucket permission / security features

    • Access Control List
    • Bucket Policy _ Will apply to specific bucket
    • CORS configuration

  • Websites and content delivery.

    • HTML, CSS, Javascript
    • Media (audio, media, movies)
    • Object versioning
      • Default versioning is turned off for new buckets.
      • When versioning is enabled, unique version number is added
      • When you delete, a delete marker is added.
      • When you undelete, original file gets back.
    • MFA delete - allows you to enforce to have MFA token to delete an object or when changing the versioning state of bucket.


  • Presigned URLs

    • aws s3 pressing s3://ac-catpics/oops.jpg 
This will generate presigned URL with necessary security credentials.
    • URL is basically accesses the website as the creator (identify).

  • Cross region replication (CRR)

    • One way replication form src bucket to destination bucket in different region
    • Versioning needs to be enabled on both buckets for CRR
    • You need to allocate a role to copy the objects from source bucket and copy TO destination bucket.
    • SSE-C encryption is not supported.
    • SSE-S3 encryption is supported.
    • SSE-KMS encryption is supported.

S3- Glacier

(It’s also storage class and service in itself).

  • Vault = Bucket
  • Archive = Object
  • 11-9’s durability.
  • Encryption in vault is ON by default and can’t turn it OFF.
  • It’s offline cold storage (May not be attached to server).
  • Need to put a job request to download the archives.
    • Standard period of time is get the archive is 3-5 hours.
    • If there is an emergency, expedited attention will be 1-5 mins (more expensive)
    • Bulk retrieval will take 5-12 hours. 
 Storage Gateway
  • Connects local data center software appliances to cloud based AWS s3.
  • VMWare or Hyper-V
  • Store local data in S3.
  • Encryption at rest and transit
Clone this wiki locally