Amazon S3 for Data Engineers

Buckets and keys, data-lake layout, CLI and boto3, storage classes and lifecycle, security and encryption, versioning and Object Lock, event notifications, performance and cost.

1. What Amazon S3 Is

Amazon S3 (Simple Storage Service) is object storage: you store objects (files plus metadata) in buckets, addressed by a key such as raw/transactions/dt=2026-10-26/part-0001.parquet. It is the foundation of almost every AWS data lake: cheap, highly durable (designed for 11 nines), and read directly by Athena, Glue, EMR, Redshift Spectrum, SageMaker and Databricks.

2. Data Lake Layout

s3://acme-lake-prod/
├── raw/  # landing zone: files exactly as received (immutable)
│   └── transactions/source=upi/ingest_date=2026-10-26/upi_20261026_0001.json.gz
├── bronze/  # same data in an open table format (Delta / Iceberg), append-only
├── silver/                   # cleaned, deduplicated, conformed
│   └── transactions/dt=2026-10-26/part-00000.snappy.parquet
├── gold/                     # business aggregates, star schemas
└── _quarantine/              # records that failed validation

s3://acme-lake-prod-logs/  # access logs and query results in a separate bucket

• Separate buckets (or at least prefixes) per zone and environment, so permissions, encryption keys, lifecycle and replication can differ.
• Hive-style partitions (dt=2026-10-26/) on low-cardinality columns used in filters; Athena, Glue and Spark prune them automatically.
• File sizes of roughly 128 MB to 1 GB, columnar (Parquet/ORC) and compressed (Snappy/ZSTD). Millions of tiny files are the most common S3 performance problem.
• Never overwrite raw data; keep it as the replay point for every downstream layer.

3. AWS CLI Essentials

aws s3 mb s3://acme-lake-dev --region ap-south-1
aws s3 ls s3://acme-lake-dev/raw/transactions/ --recursive --human-readable --summarize
aws s3 cp ./txn.json.gz s3://acme-lake-dev/raw/transactions/ingest_date=2026-10-26/
aws s3 cp s3://acme-lake-dev/silver/ ./local/ --recursive --exclude "*" --include "*.parquet"
aws s3 sync ./exports/ s3://acme-lake-dev/raw/exports/ --delete  # mirror a folder
aws s3 mv s3://acme-lake-dev/raw/a.csv s3://acme-lake-dev/archive/a.csv
aws s3 rm s3://acme-lake-dev/tmp/ --recursive

aws s3api head-object --bucket acme-lake-dev --key raw/transactions/x.json.gz  # size, ETag, metadata
aws s3api list-object-versions --bucket acme-lake-dev --prefix silver/customers/
aws s3 presign s3://acme-lake-dev/reports/daily.csv --expires-in 3600  # temporary share link

4. S3 with Python (boto3)

import boto3
from boto3.s3.transfer import TransferConfig

s3 = boto3.client("s3")  # credentials from the IAM role / environment, never hard-coded

# upload with multipart for large files (parallel parts, automatic retries)
cfg = TransferConfig(multipart_threshold=64 * 1024 * 1024, max_concurrency=8)
s3.upload_file("big_extract.parquet", "acme-lake-dev", "raw/extracts/big_extract.parquet",
               Config=cfg, ExtraArgs={"ServerSideEncryption": "aws:kms"})

# list every object under a prefix (a single call returns at most 1,000 keys)
paginator = s3.get_paginator("list_objects_v2")
keys = [obj["Key"]
        for page in paginator.paginate(Bucket="acme-lake-dev", Prefix="raw/transactions/")
        for obj in page.get("Contents", [])]

# stream a large object line by line without loading it all into memory
body = s3.get_object(Bucket="acme-lake-dev", Key="raw/events/2026-10-26.jsonl")["Body"]
for line in body.iter_lines():
    process(line)

# write small results / markers
s3.put_object(Bucket="acme-lake-dev", Key="raw/transactions/dt=2026-10-26/_SUCCESS", Body=b"")

url = s3.generate_presigned_url("get_object", Params={"Bucket": "acme-lake-dev", "Key": "reports/daily.csv"},
                                ExpiresIn=3600)

# pandas / Parquet straight from S3 with the AWS SDK for pandas (awswrangler)
import awswrangler as wr

df = wr.s3.read_parquet("s3://acme-lake-dev/silver/transactions/", dataset=True,
                        partition_filter=lambda p: p["dt"] == "2026-10-26")
wr.s3.to_parquet(df, path="s3://acme-lake-dev/gold/daily_summary/", dataset=True,
                 partition_cols=["dt"], mode="overwrite_partitions")

5. Storage Classes & Lifecycle

{
  "Rules": [
    {
      "ID": "raw-tiering",
      "Filter": { "Prefix": "raw/" },
      "Status": "Enabled",
      "Transitions": [
        { "Days": 30,  "StorageClass": "STANDARD_IA" },
        { "Days": 180, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 2555 },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 30 },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

aws s3api put-bucket-lifecycle-configuration --bucket acme-lake-prod --lifecycle-configuration file://lifecycle.json

Always add AbortIncompleteMultipartUpload: failed multipart uploads leave invisible parts that you keep paying for.

6. Security: Access, Encryption & Governance

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": ["arn:aws:s3:::acme-lake-prod", "arn:aws:s3:::acme-lake-prod/*"],
      "Condition": { "Bool": { "aws:SecureTransport": "false" } }
    },
    {
      "Sid": "DenyWrongKmsKey",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::acme-lake-prod/*",
      "Condition": {
        "StringNotEqualsIfExists": {
          "s3:x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:ap-south-1:111122223333:key/1111-2222"
        }
      }
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "ListSilverOnly", "Effect": "Allow", "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::acme-lake-prod",
      "Condition": { "StringLike": { "s3:prefix": ["silver/*"] } } },
    { "Sid": "ReadWriteSilver", "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::acme-lake-prod/silver/*" }
  ]
}

The second policy is an IAM (identity) policy for a Glue job role: it can list and change only the silver prefix. Note that ListBucket applies to the bucket ARN, while object actions apply to bucket/* ARNs: a common source of AccessDenied errors.

7. Versioning, Object Lock & Replication

• Versioning: a delete adds a delete marker and overwrites keep old versions, so you can recover. Pair it with a lifecycle rule that expires noncurrent versions.
• Object Lock (WORM): retention periods in governance mode (privileged users can override) or compliance mode (nobody, including root, can delete until expiry). Used for audit, AML and regulatory records. Legal holds block deletion indefinitely.
• Replication: Cross-Region (CRR) for disaster recovery or data residency copies, Same-Region (SRR) for log aggregation or separate accounts. Needs versioning on both buckets; Replication Time Control adds an SLA.
• Backups: AWS Backup can protect S3 with point-in-time restore.

8. Event Notifications

S3 can announce new or deleted objects to Lambda, SQS, SNS, or EventBridge (which allows richer filtering and many targets). This is how file-arrival pipelines start.

# send all events to EventBridge, then route with rules
aws s3api put-bucket-notification-configuration --bucket acme-lake-prod \
  --notification-configuration '{"EventBridgeConfiguration": {}}'

# EventBridge rule pattern: new .csv files under raw/partner/
{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": { "name": ["acme-lake-prod"] },
    "object": { "key": [{ "wildcard": "raw/partner/*.csv" }] }
  }
}

• Notifications are at-least-once: occasionally duplicated and not guaranteed in order. Make processing idempotent (track processed keys or ETags).
• For bursts of thousands of files, put SQS between S3 and the consumer to buffer and control concurrency.
• Avoid recursive loops: a function triggered by a prefix must not write back into the same prefix.

9. Performance & Cost

S3 Tables offer table buckets with fully managed Apache Iceberg tables (automatic compaction and snapshot management); see the Apache Iceberg guide.

10. Common Interview Questions