Cloud Platforms
Storage, compute, orchestration, security and governance on AWS and Azure, with service maps and reference architectures.
19 guides
Each guide is a complete reference with explanations, code and diagrams.
Overviews
AWS Overview
Compute, storage, IAM, networking, SAS-to-AWS migration, the data services map and exam notes.
AWSIAMArchitectureRead guideAzure
ADLS Gen2, Data Factory, Azure Databricks, Synapse, Fabric, Event Hubs, Cosmos DB and Bicep.
ADFADLSFabricRead guideStorage & Transfer
Amazon S3
Data-lake layout, CLI and boto3, storage classes and lifecycle, bucket policies and encryption, Object Lock, events and performance.
S3LifecycleSecurityRead guideAWS File Transfer
Transfer Family SFTP/FTPS/AS2 for partners, DataSync migrations, Storage Gateway, offline and accelerated transfer.
SFTPDataSyncPartnersRead guideAWS DMS
Full load and CDC from databases, source prerequisites, table mappings, CDC to S3 and MERGE into lake tables.
CDCMigrationReplicationRead guideProcessing & ETL
AWS Glue
Data Catalog, crawlers, a full PySpark job, bookmarks, Iceberg writes, parallel JDBC, Data Quality and tuning.
GlueCatalogETLRead guideAmazon EMR
EMR on EC2, Serverless and EKS, node types and fleets, clusters and steps, Glue Catalog, tuning and security.
EMRSparkServerlessRead guideAWS Lambda
Limits, invocation models, S3/SQS/Kinesis handlers, packaging, secrets and VPC, idempotency and anti-patterns.
LambdaServerlessEventsRead guideAnalytics & Warehousing
Amazon Athena
Tables, partition projection, CTAS/UNLOAD, Iceberg tables, cost optimisation, Python access and troubleshooting.
AthenaSQLIcebergRead guideAmazon Redshift
Architecture, dist and sort keys, COPY/UNLOAD, MERGE and SCD2, Spectrum, zero-ETL, streaming ingestion and the Data API.
RedshiftWarehouseMERGERead guideDatabases
Amazon RDS & Aurora
Engines, Multi-AZ and replicas, Aurora architecture, security, extracting for the lake and CDC prerequisites.
RDSAuroraPostgreSQLRead guideAmazon DynamoDB
Keys and capacity, access-pattern modelling, conditional writes and watermarks, streams, exports and TTL.
DynamoDBNoSQLStreamsRead guideOther AWS Databases
DocumentDB, ElastiCache and MemoryDB, Neptune, Timestream, Keyspaces, OpenSearch, and how to choose.
GraphCacheSearchRead guideStreaming & Messaging
Amazon Kinesis
Data Streams shards, producers and consumers, Firehose delivery with dynamic partitioning and Parquet conversion.
KinesisFirehoseStreamingRead guideAmazon MSK
Managed Kafka options, IAM authentication, Python and Spark clients, MSK Connect and operations.
KafkaMSKIAMRead guideAmazon SQS
Standard vs FIFO, visibility timeout, producers and consumers, DLQs and redrive, Lambda integration and patterns.
SQSQueuesDLQRead guideAmazon SNS & EventBridge
Fan-out with filter policies, pipeline alerting, EventBridge rules, Scheduler and Pipes, messaging comparison.
SNSEventBridgeAlertsRead guideOrchestration & IaC
AWS Step Functions
A full ETL state machine, .sync and callbacks, data passing, Map and Distributed Map, retries and monitoring.
OrchestrationASLWorkflowsRead guideAWS CloudFormation
Templates and change sets, a full data-lake stack, intrinsic functions, safe deployments, CDK and SAM.
IaCCDKTemplatesRead guide