AWS Reference

Compute, storage classes, data lakes, and IAM — condensed from Solutions Architect and Data Engineer certification material.

1. Compute

• EC2 — resizable virtual servers, the base compute unit.
• EC2 Auto Scaling — adds/removes instances to match demand against a scaling policy.
• Elastic Load Balancing — distributes traffic across instances/targets, works with Auto Scaling for resilience.
• EKS — managed Kubernetes for container orchestration.
• AWS Batch — runs batch computing jobs at any scale without managing the underlying infrastructure.

2. Storage

S3 is highly available, highly durable, cost-effective and widely accessible — the default for data lakes, backups, and static hosting. Its storage classes trade cost against retrieval speed:

Block & file storage for EC2

Rule of thumb: shared access across instances → EFS. Single-instance persistent disk → EBS. Object storage accessible to many clients over the internet → S3.

3. Lifecycle & Versioning

• Lifecycle rules — automatically transition objects between storage classes, or delete them, after a set time. The tool for "keep accessible for 90 days, then archive/delete."
• Versioning — keeps prior versions of an object so you can recover from accidental overwrite or delete. Does not move data between storage classes.
• Access control — IAM identity-based policies, S3 bucket policies (resource-based), Access Control Lists, S3 Block Public Access, and CORS.

4. Data Lakes

A data lake supports both structured and unstructured data in one place, typically S3-backed. Core services:

• AWS Glue — crawlers build a data catalog by inferring schema from files; Glue jobs transform data (formatting, partitioning, compression) for efficient query.
• AWS Lake Formation — builds and secures a data lake on top of S3 with a centralized permission model; automates ingestion with blueprints and workflows; includes FindMatches for record matching.
• Amazon Athena — serverless SQL queries directly over data in S3/the Glue catalog.
• Amazon QuickSight — visualization layer on top of the lake.
• Amazon Redshift — managed data warehouse for structured, curated analytics workloads at scale.

Typical flow: ingest raw data to S3 → crawl/catalog with Glue → transform with Glue jobs → secure with Lake Formation → query with Athena or load curated data into Redshift → visualize with QuickSight.

AWS Glue in practice

A crawler scans an S3 path, infers schema and file format, and writes table definitions into the Glue Data Catalog — the metastore Athena, Redshift Spectrum and EMR all read from. Point crawlers at specific prefixes (e.g. one per dataset) rather than a whole bucket, since a crawler can otherwise create unwanted tables from unrelated subfolders.

import boto3

glue = boto3.client("glue")

# Create and run a crawler
glue.create_crawler(
    Name="sales-crawler",
    Role="AWSGlueServiceRole",
    DatabaseName="sales_db",
    Targets={"S3Targets": [{"Path": "s3://my-bucket/sales/"}]}
)
glue.start_crawler(Name="sales-crawler")

# Inspect what the crawler cataloged
tables = glue.get_tables(DatabaseName="sales_db")
for t in tables["TableList"]:
    print(t["Name"], t["StorageDescriptor"]["Location"])

A Glue ETL job is a managed Spark (or Python shell) script that reads from the catalog, transforms, and writes back to S3 in an analytics-friendly format:

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from pyspark.context import SparkContext

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session

# Read from the Data Catalog
datasource = glueContext.create_dynamic_frame.from_catalog(
    database="sales_db", table_name="raw_orders"
)

# Transform: drop nulls, rename a field
cleaned = datasource.drop_fields(["temp_col"]) \
                     .rename_field("cust_id", "customer_id")

# Write back as partitioned Parquet
glueContext.write_dynamic_frame.from_options(
    frame=cleaned,
    connection_type="s3",
    connection_options={"path": "s3://my-bucket/curated/orders/", "partitionKeys": ["order_year"]},
    format="parquet"
)

Query the cataloged table straight away with Athena:

SELECT region, SUM(amount) AS total_sales
FROM sales_db.curated_orders
WHERE order_year = 2026
GROUP BY region
ORDER BY total_sales DESC;

Remember the schema-evolution rule from §8: keep new CSV/TSV columns appended at the end (Athena reads by index); Parquet tolerates reordering/removal but not renaming (Athena reads by name). Re-run the crawler, or use its schedule, whenever upstream files change shape.

5. IAM & Access

• Users, groups, roles — roles are assumed (by services or federated users) rather than tied to permanent credentials — prefer roles over long-lived access keys.
• Policies — JSON documents granting or denying actions on resources; identity-based (attached to a user/role) or resource-based (attached to the resource, e.g. an S3 bucket policy).
• Least privilege — grant only the actions/resources actually needed.

6. Global Infrastructure

• Availability Zone (AZ) — one or more discrete data centers with redundant power and networking; a common misconception is that an AZ equals a single data center — it's often several, physically close together.
• Region — a cluster of AZs (always at least two) in one geographic area, linked by low-latency private fiber but isolated from each other so a single AZ failure doesn't take down the Region. Regions operate fully independently for latency, compliance and data-residency reasons.
• Edge Locations — CloudFront's content-delivery points of presence, far more numerous than Regions, caching content close to end users.
• Regional Edge Caches — a caching layer between origin servers and edge locations, holding content less frequently accessed than what sits at the edge.

Best practice: spread workloads across at least two AZs in a Region for high availability — RDS Multi-AZ, for example, keeps a synchronously replicated standby in a second AZ. Data is not copied between Regions unless you explicitly configure it to be, which is what keeps data-residency commitments intact.

7. Cloud Economics

8. Data Lake vs Data Warehouse

A warehouse feeds two needs: operational (one record at a time, no long history) and analytical (thousands of records at once, fast queries, historical context) — the ETL pipeline moves data from source systems through a staging area into the warehouse's core, then out to department-specific data marts.

Glue Catalog schema evolution depends on file format: Athena reads CSV/TSV by column index, so columns must stay in order (new columns only appended at the end; removing a column breaks the mapping) but can be freely renamed in the catalog. Parquet is read by column name, so columns can be reordered, added, or removed safely — but renaming a column in the catalog orphans the old data instead of finding it.

9. Networking (VPC)

• VPC — an isolated virtual network you control the IP range, subnets and routing for.
• Subnets — a public subnet routes to an Internet Gateway; a private subnet does not, and reaches the internet (outbound only) via a NAT Gateway sitting in a public subnet.
• Route tables — per-subnet rules deciding where traffic goes; the public/private distinction is really just which route table a subnet uses.
• Security Groups — stateful, instance-level firewalls: allow rules only (no explicit deny), and a response to allowed inbound traffic is automatically allowed out.
• Network ACLs — stateless, subnet-level firewalls: support explicit allow and deny rules, evaluated in numbered order; because they're stateless, return traffic needs its own rule.
• VPC Peering / Transit Gateway — connect VPCs to each other; Transit Gateway scales this to many VPCs through one hub instead of a peering mesh.

10. Databases

Multi-AZ keeps a synchronously replicated standby in a second AZ for failover; read replicas are asynchronous copies that offload read traffic and can be promoted, but aren't meant for automatic failover.

11. Serverless (Lambda)

AWS Lambda runs code in response to an event without provisioning servers — billed per invocation and per millisecond of execution.

import json

def handler(event, context):
    for record in event["Records"]:
        body = json.loads(record["body"])
        process(body)
    return {"statusCode": 200}

Common triggers: S3 object-created events, SQS/SNS messages, API Gateway requests, Kinesis/DynamoDB streams, and scheduled EventBridge rules. Watch for cold starts (first invocation after idle time pays initialization cost) and the default 15-minute max execution time — longer jobs belong on ECS/Fargate, Batch, or EMR instead.

12. Monitoring

• CloudWatch — metrics, logs, and alarms; the default place resource utilization (CPU, latency, error rate) and custom application metrics both live.
• CloudTrail — an audit log of every API call made against the account — who did what, when, from where. Security and compliance's primary tool, not operational monitoring.
• AWS Config — tracks configuration changes to resources over time and can evaluate them against compliance rules.

Exam shorthand: "who changed this security group" → CloudTrail. "Alert when CPU exceeds 80%" → CloudWatch. "Is this resource compliant with our tagging policy" → Config.

13. Well-Architected Framework

14. Exam Domains & Sample Questions

The Solutions Architect Associate (SAA-C02) exam weights four domains: Resilient Architectures (30%), High-Performing Architectures (28%), Secure Applications (24%), Cost-Optimized Architectures (18%). Passing score is 720 of 1,000; unanswered questions count as incorrect, so always guess rather than skip.

15. SAS to AWS Migration

Phases

• 1. Discovery: scan code (PROCs, macros, inputs, outputs), build lineage, use logs to see what actually runs, score complexity, retire dead code (often 20–40%), plan migration waves.
• 2. Target design: S3 zones, formats, Glue vs EMR, IAM / Lake Formation / KMS, environments, a shared conversion library.
• 3. Data migration: pandas.read_sas / pyreadstat or spark-sas7bdat; DataSync or Snowball for bulk files; DMS for databases.
• 4. Code conversion: manual, automated converters or AI-assisted — always reviewed and reconciled.
• 5. Reconciliation: run SAS and PySpark on the same inputs; compare schema, counts, aggregates, nulls and row hashes within tolerances; record results in an audit table.
• 6. Orchestration: Step Functions or MWAA, EventBridge, Lambda triggers, SNS, CloudWatch.
• 7. Reports and models: rebuild in QuickSight / Athena / SageMaker and re-validate.
• 8. Cut-over: parallel run, cut over and decommission wave by wave with business sign-off.

SAS-specific traps

Conversion patterns

w = Window.partitionBy("account_id").orderBy("txn_date")
acct_summary = (txn
    .withColumn("running_total",
                F.sum("amount").over(w.rowsBetween(Window.unboundedPreceding, 0)))
    .withColumn("rn", F.row_number().over(w.orderBy(F.col("txn_date").desc())))
    .filter("rn = 1").drop("rn"))      # SAS RETAIN + LAST. by account

Typical causes of mismatches: missing-value semantics, rounding, implicit sort order, many-to-many MERGE, truncation, timezone offsets. Define "done" per program as reconciliation PASS + code review + scheduled.

16. AWS Services for Data Engineering

Reference pipeline

On-prem files ──DataSync──► S3 raw/      Oracle ──DMS (full load + CDC)──► S3 raw/
EventBridge → Step Functions:
  1 Glue job raw → curated
  2 Data quality + reconciliation   (FAIL → SNS, stop)
  3 Redshift COPY (star schema, SCD2 MERGE)
  4 Update catalog partitions → Athena
  5 SNS: done
Cross-cutting: Lake Formation · KMS · CloudWatch · QuickSight

Four-week study plan

Set a budget alert, prefer serverless options and delete resources after each session. Relevant certification: AWS Certified Data Engineer – Associate (DEA-C01).

17. Workloads Longer Than Lambda's 15 Minutes

The 15-minute Lambda limit is fixed. Keep Lambda for short, event-driven steps and either hand long work to another service or split it into smaller units.

"RunSettlementLoad": {
  "Type": "Task",
  "Resource": "arn:aws:states:::glue:startJobRun.sync",
  "Parameters": {
    "JobName": "settlement-load",
    "Arguments": { "--business_date.$": "$.business_date" }
  },
  "Retry": [{ "ErrorEquals": ["States.ALL"], "MaxAttempts": 2, "IntervalSeconds": 300 }],
  "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "NotifyFailure" }],
  "Next": "Reconcile"
}

Example: a 20 GB settlement file lands → EventBridge → Lambda validates it (seconds) → Step Functions runs Glue/EMR/Databricks with .sync (40 min) → Lambda writes a reconciliation record → SNS. Fan-out workers must be idempotent; cap concurrency to protect downstream systems; never make a Lambda call itself in a loop (AWS stops recursive loops), so loop with Step Functions instead.