Data Architecture Patterns

Lambda and Kappa in depth with worked examples, a decision guide, and the patterns around them: medallion, lakehouse, data mesh, CDC and outbox, ETL vs ELT, streaming reliability, serving, and reference architectures.

1. How to Choose an Architecture

A data architecture pattern is a proven way to arrange ingestion, storage, processing and serving. No pattern is best in general; you choose by answering a few questions about the workload:

Interview tip: state the requirements first ("sub-second alerts, daily regulatory reports, 2 TB/day, late settlement files"), then pick the pattern and explain the trade-off you are accepting.

2. Lambda Architecture

Lambda (Nathan Marz, around 2011) runs two pipelines over the same immutable data: a batch layer that periodically recomputes complete, accurate results from all history, and a speed layer that processes recent events immediately to fill the gap until the next batch run. A serving layer merges both views for queries.

                       ┌──────────────────── BATCH LAYER ─────────────────────┐
                       │ master dataset (immutable, append-only: S3 / Delta)    │
 sources ──► Kafka ──┬─►│ → nightly recompute from ALL history (Spark)          │──► batch views ──┐
 (events,     (log)  │  └────────────────────────────────────────────────────────┘   (complete,     │
  files)             │                                                                 correct)      ├──► SERVING
                     │  ┌──────────────────── SPEED LAYER ─────────────────────┐                   │    LAYER
                     └─►│ stream processing (Flink / Spark Structured Streaming)│──► real-time ─────┘  (merge views,
                        │ only data since the last batch run, low latency        │    views              answer queries)
                        └────────────────────────────────────────────────────────┘    (approximate,
                                                                                         recent only)

Worked example: daily transaction totals per account

Requirement: dashboards need today's running totals within seconds, but finance needs exact daily totals, including late settlement corrections. The speed layer keeps today's totals fresh:

from pyspark.sql import functions as F, types as T
from delta.tables import DeltaTable

schema = T.StructType([
    T.StructField("txn_id", T.StringType()),
    T.StructField("account_id", T.StringType()),
    T.StructField("amount", T.DecimalType(18, 2)),
    T.StructField("event_ts", T.TimestampType()),
])

events = (spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "kafka:9092")
    .option("subscribe", "txn.events")
    .option("startingOffsets", "latest")
    .load()
    .select(F.from_json(F.col("value").cast("string"), schema).alias("t"))
    .select("t.*"))

running = (events
    .withWatermark("event_ts", "10 minutes")
    .groupBy("account_id", F.to_date("event_ts").alias("txn_date"))
    .agg(F.sum("amount").alias("total_amount"), F.count("*").alias("txn_count")))

def upsert_speed(batch_df, batch_id):  # MERGE so totals update in place
    (DeltaTable.forName(spark, "speed.daily_txn_totals").alias("t")
        .merge(batch_df.alias("s"), "t.account_id = s.account_id AND t.txn_date = s.txn_date")
        .whenMatchedUpdateAll().whenNotMatchedInsertAll().execute())

(running.writeStream
    .outputMode("update")
    .foreachBatch(upsert_speed)
    .option("checkpointLocation", "/chk/speed_daily_txn_totals")
    .trigger(processingTime="30 seconds")
    .start())

The batch layer recomputes exact totals for completed days from the master dataset every night, including late and corrected records:

exact = (spark.read.table("bronze.transactions")  # immutable master dataset
    .dropDuplicates(["txn_id"])  # batch can afford a global dedupe
    .join(spark.read.table("bronze.settlement_adjustments"), "txn_id", "left")
    .withColumn("final_amount", F.coalesce("adjusted_amount", "amount"))
    .where(F.col("txn_date") < F.current_date())  # complete days only
    .groupBy("account_id", "txn_date")
    .agg(F.sum("final_amount").alias("total_amount"), F.count("*").alias("txn_count")))

exact.write.mode("overwrite").saveAsTable("batch.daily_txn_totals")  # recompute, don't patch

The serving layer answers queries from the batch view where it exists and from the speed view only for days the batch has not covered yet:

CREATE OR REPLACE VIEW serving.daily_txn_totals AS
SELECT account_id, txn_date, total_amount, txn_count, 'batch' AS source
FROM batch.daily_txn_totals
UNION ALL
SELECT account_id, txn_date, total_amount, txn_count, 'speed' AS source
FROM speed.daily_txn_totals
WHERE txn_date > (SELECT MAX(txn_date) FROM batch.daily_txn_totals);   -- only uncovered days

-- daily reconciliation: how far did the fast path drift from the exact numbers?
SELECT b.txn_date,
       SUM(b.total_amount)                         AS batch_total,
       SUM(s.total_amount)                         AS speed_total,
       SUM(s.total_amount) - SUM(b.total_amount)   AS drift
FROM batch.daily_txn_totals b
JOIN speed.daily_txn_totals s USING (account_id, txn_date)
WHERE b.txn_date = current_date() - INTERVAL 1 DAY
GROUP BY b.txn_date;

Reduce duplicated logic by sharing it: keep rules in SQL or a shared Python library used by both layers, or let one engine (e.g. Spark) run both paths. Banking fraud platforms often keep Lambda deliberately: Flink for millisecond decisions plus a nightly lakehouse recomputation for regulatory accuracy (see the Kafka & Kinesis guide for that setup).

3. Kappa Architecture

Kappa (Jay Kreps, 2014) removes the batch layer: everything is a stream. All data goes into a durable, replayable log, one stream-processing codebase produces the serving tables, and reprocessing means replaying the log through a new version of the job.

 sources ──► Kafka / Kinesis / Delta bronze (replayable log, long retention) ──► stream processor v1 ──► serving tables v1
                                         │                                                               ▲ readers
                                         └── replay from offset 0 ──► stream processor v2 ──► serving tables v2
                                                                       (new logic)            switch readers when
                                                                                              v2 has caught up

Worked example: one job, versioned outputs

from pyspark.sql import functions as F

def build_totals(version: str, starting_offsets: str):
    events = (spark.readStream.format("kafka")
        .option("kafka.bootstrap.servers", "kafka:9092")
        .option("subscribe", "txn.events")
        .option("startingOffsets", starting_offsets)
        .load()
        .select(F.from_json(F.col("value").cast("string"), schema).alias("t"))
        .select("t.*")
        .dropDuplicates(["txn_id"]))  # stream dedupe (bounded by the watermark)

    totals = (events
        .withWatermark("event_ts", "2 hours")
        .groupBy(F.window("event_ts", "1 day").alias("w"), "account_id")
        .agg(F.sum("amount").alias("total_amount")))

    return (totals.writeStream
        .outputMode("append")  # windows emitted once final
        .option("checkpointLocation", f"/chk/txn_totals_{version}")
        .toTable(f"gold.txn_totals_{version}"))

v1 = build_totals("v1", "latest")       # running in production

# Logic changed (e.g. new rounding rule)? Deploy v2 alongside v1 and replay the whole log:
v2 = build_totals("v2", "earliest")

-- once v2 has caught up to the head of the log and reconciles with v1:
CREATE OR REPLACE VIEW gold.txn_totals AS SELECT * FROM gold.txn_totals_v2;   -- switch readers
-- then stop v1 and drop gold.txn_totals_v1

Kappa on a lakehouse

Keeping months of data in Kafka is expensive. A common modern variant uses an append-only Delta or Iceberg bronze table as the replayable log: Kafka feeds bronze, and downstream streams read the table, so replay is just a new stream from the beginning of bronze.

(spark.readStream.table("bronze.transactions")  # the table IS the replayable log
    .withWatermark("event_ts", "2 hours")
    .groupBy(F.window("event_ts", "1 day"), "account_id")
    .agg(F.sum("amount").alias("total_amount"))
    .writeStream.outputMode("append")
    .option("checkpointLocation", "/chk/gold_totals_v3")
    .trigger(availableNow=True)  # same code runs incrementally OR as a "batch"
    .toTable("gold.txn_totals_v3"))

4. Lambda vs Kappa: Side by Side

Decision guide

Modern lakehouses blur the line: Delta and Iceberg tables can be read as batch or as stream, so the same code can serve both latency needs. Many teams today run "Kappa on the lakehouse" and add a separate Flink path only where milliseconds matter.

5. Medallion (Bronze / Silver / Gold)

The medallion pattern organises a lakehouse into layers of increasing quality. It is orthogonal to Lambda/Kappa: either can feed and maintain medallion layers.

# bronze → silver: dedupe, type, validate, upsert
silver = (spark.readStream.table("bronze.orders")
    .withColumn("amount", F.col("amount").cast("decimal(18,2)"))
    .filter(F.col("order_id").isNotNull())
    .dropDuplicatesWithinWatermark(["order_id"])  # Spark 3.5+
    .withWatermark("updated_at", "1 hour"))

# silver → gold: business aggregate for dashboards
gold = (spark.read.table("silver.orders")
    .groupBy("order_date", "region")
    .agg(F.sum("amount").alias("revenue"), F.countDistinct("customer_id").alias("customers")))

See the Data Lake and Spark & Databricks guides for full medallion pipelines with Auto Loader, MERGE and SCD2.

6. Warehouse, Lake, Lakehouse

The lakehouse arose to stop running a lake for ML and a separate warehouse for BI with copies and pipelines between them. Many organisations still combine a lakehouse with a warehouse for specific BI needs.

7. Data Mesh & Data Fabric

Data mesh (Zhamak Dehghani, 2019) is an organisational pattern for large companies where one central data team has become a bottleneck. It rests on four principles:

# a data product contract (often kept in Git next to the pipeline code)
dataProduct: payments.daily_settlements
owner: payments-data-team@acme.com
description: One row per merchant per business date with settled amounts.
output:
  table: payments_prod.gold.daily_settlements
  format: delta
schema:
  - { name: merchant_id,   type: string,        nullable: false }
  - { name: business_date, type: date,          nullable: false }
  - { name: settled_amount, type: decimal(18,2), nullable: false }
quality:
  - freshness: available by 06:00 IST on business days
  - unique: [merchant_id, business_date]
  - row_count_change_vs_previous_day: "< 30%"
classification: confidential
access: request via data catalog; approval by product owner

Data fabric is a technology approach: an integration layer driven by active metadata (catalog, lineage, semantics) that connects data across many systems, often virtualising access rather than copying data. Mesh changes who owns data; fabric changes how it is connected. They can coexist.

8. Change Data Capture & the Outbox Pattern

{
  "name": "orders-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "database.hostname": "orders-db.internal",
    "database.port": "5432",
    "database.user": "cdc_reader",
    "database.password": "${file:/opt/secrets/orders-db.properties:password}",
    "database.dbname": "orders",
    "topic.prefix": "orders",
    "table.include.list": "public.orders,public.order_items"
  }
}

Each change becomes a Kafka event with before, after and op (c = create, u = update, d = delete, r = snapshot read). Apply them to the lake with MERGE, keeping the latest change per key:

from pyspark.sql import Window

def apply_cdc(batch_df, batch_id):
    latest = (batch_df
        .withColumn("rn", F.row_number().over(
            Window.partitionBy("order_id").orderBy(F.col("source_lsn").desc())))
        .filter("rn = 1").drop("rn"))
    (DeltaTable.forName(spark, "silver.orders").alias("t")
        .merge(latest.alias("s"), "t.order_id = s.order_id")
        .whenMatchedDelete(condition="s.op = 'd'")
        .whenMatchedUpdateAll(condition="s.op IN ('u', 'c', 'r')")
        .whenNotMatchedInsertAll(condition="s.op <> 'd'")
        .execute())

The transactional outbox pattern

A service must update its database and publish an event. Doing both separately risks one succeeding and the other failing ("dual write"). Instead, write the event to an outbox table in the same database transaction, and let CDC publish it:

BEGIN;
  UPDATE accounts SET balance = balance - 500 WHERE account_id = 'ACC1';
  INSERT INTO outbox (event_id, aggregate_id, event_type, payload, created_at)
  VALUES (gen_random_uuid(), 'ACC1', 'FundsDebited', '{"amount": 500}', now());
COMMIT;
-- Debezium reads the outbox rows from the log and publishes them to Kafka:
-- the event is published if and only if the transaction committed.

9. ETL, ELT & Reverse ETL

10. Streaming Reliability Patterns

parsed = raw.select(
    F.col("value").cast("string").alias("raw_value"),
    F.from_json(F.col("value").cast("string"), schema).alias("t"))

good = parsed.filter("t.txn_id IS NOT NULL AND t.amount IS NOT NULL").select("t.*")
bad = (parsed.filter("t.txn_id IS NULL OR t.amount IS NULL")
          .withColumn("error", F.lit("missing txn_id or amount"))
          .withColumn("quarantined_at", F.current_timestamp()))

good.writeStream.option("checkpointLocation", "/chk/silver_txn").toTable("silver.transactions")
bad.writeStream.option("checkpointLocation", "/chk/dlq_txn").toTable("quarantine.transactions")  # DLQ

11. Serving Patterns

12. Reference Architectures

Real-time fraud detection (Lambda, bank)

 cards / UPI / ATM ──► Kafka (txn events, keyed by account)
                         ├──► Flink: per-account velocity + rules + model (ms) ──► decision API, alerts topic, Redis
                         └──► Delta bronze ──► nightly Databricks: silver (dedupe, settlements) ──► gold (exact totals,
                                                regulatory reports, model training) ──► reconciled against Flink outputs

E-commerce analytics (Kappa on a lakehouse)

 clickstream, orders (CDC via Debezium) ──► Kafka ──► Delta bronze (append-only, full history)
   ──► Spark Structured Streaming: silver sessions, orders (MERGE) ──► gold funnels, revenue (streaming tables)
   ──► SQL warehouse dashboards (refresh every minute) · replay bronze into _v2 tables when logic changes

Enterprise analytics (batch ELT + medallion)

 ERP / CRM / HR databases ──► nightly incremental extract (ADF / Glue / Fivetran) ──► raw landing (S3 / ADLS)
   ──► Auto Loader bronze ──► dbt / Spark silver (SCD2 dimensions) ──► gold star schemas ──► Power BI / Tableau
   orchestrated by Airflow · governed by Unity Catalog / Lake Formation · data-quality gates between layers

13. Common Interview Questions