Data Engineering
Build reliable batch and streaming pipelines: Hadoop and Spark, Kafka and Flink, data warehouses and lakes, and the Delta, Iceberg and Hudi table formats.
17 guides
Each guide is a complete reference with explanations, code and diagrams.
Hadoop & HDFS
Big data fundamentals, HDFS architecture, replication and fault tolerance, Hive and its evolution.
HDFSYARNHiveRead guideSpark & Databricks
DataFrames, joins, tuning and salting, cluster sizing, Delta, Auto Loader, Lakeflow and production Databricks.
PySparkDatabricksTuningRead guideKafka & Kinesis
Topics, partitions and offsets, NiFi, Lambda vs Kappa, watermarks and the Kafka engineer role.
KafkaKinesisNiFiRead guideData Architecture Patterns
Lambda vs Kappa with worked examples, medallion, lakehouse, data mesh, CDC and outbox, streaming reliability and reference architectures.
LambdaKappaData MeshRead guideEnterprise Data Architect
Architecture styles, table formats, governance, security and lineage across clouds, plus implementing lakes, warehouses, lakehouses and data mesh, and modernising legacy systems.
ArchitectureGovernanceModernisationRead guidePyFlink
Flink SQL and the DataStream API in Python, state, checkpoints, watermarks and deployment.
FlinkStreamingStateRead guideApache Airflow
Workflows as code: DAGs, TaskFlow, scheduling and data intervals, sensors, XCom, dynamic tasks, assets and orchestrating Spark and Databricks.
AirflowDAGsOrchestrationRead guideData Warehouse
OLTP vs OLAP, star schemas, SCDs, normal forms, cardinality and Redshift SQL.
ModellingSCD2RedshiftRead guideData Modelling
Conceptual to physical models, star schemas, building a model in erwin, and running its DDL with PySpark on Databricks.
ModellingerwinDDLRead guidedbt
SQL transformations with tests and lineage: models, snapshots, incremental loads, contracts, and running dbt on Databricks with an end-to-end retail use case.
dbtELTDatabricksRead guideData Lake
Lake architecture, Lake Formation, lakehouse on Databricks, Unity Catalog, governance and compliance.
LakehouseGovernanceUnity CatalogRead guideIncremental Load
Watermarks, CDC, file-based ingestion and merge/upsert patterns.
CDCWatermarksMERGERead guideDelta Lake
Transaction log, MERGE, time travel, schema evolution, OPTIMIZE/VACUUM and Change Data Feed.
DeltaACIDTime travelRead guideApache Iceberg
Metadata layers, catalogs, hidden partitioning, branches, CoW vs MoR and Iceberg on AWS.
IcebergCatalogsAthenaRead guideApache Hudi
CoW vs MoR, upserts, incremental queries, indexing, and a Delta vs Iceberg vs Hudi comparison.
HudiUpsertsCDCRead guideMongoDB
Document model, mongosh commands, aggregation pipeline, indexes and the Spark connector.
NoSQLAggregationIndexesRead guidePython
Core language, data structures, OOP in depth (encapsulation, abstraction, inheritance, polymorphism), files and pandas.
PythonOOPpandasRead guide