Data Lake Reference

Architecture, governance, AWS Lake Formation, and the schema-evolution rules that make or break a lake in practice.

1. What a Data Lake Is

A data lake stores files in their native format (S3), catalogs their structure as queryable tables (the Glue Data Catalog), and lets you query them with SQL (Athena) — without first forcing every source into a rigid relational schema. That combination makes it well suited to data discovery, exploratory data analysis, and machine learning, where the eventual use of the data isn't fully known up front.

• Data discovery — finding and identifying data across an organization: source, quality, structure, and its relationship to other assets. The critical first step of any data-driven project.
• Exploratory Data Analysis (EDA) — cleaning, merging, transforming and summarizing data (statistically and visually) once discovered. A lake gives engineers, analysts and ML practitioners access to raw data whose purpose isn't yet defined.

2. Vs. Relational Databases

Relational databases remain excellent for tabular, well-understood data: SQL, automatic type/integrity checks, consistency, security, and a mature ecosystem — at the cost of complexity, expense, rigid structure, and ongoing development effort. A data lake trades that rigidity for flexibility:

Most organizations need both, feeding each other — see the separate Data Warehouse reference doc for the warehouse side of this picture.

3. Reference Architecture

A data lake is not one product but an architecture assembled from several services, each covering one concern:

Organize S3 itself into zones: a raw/landing zone for unmodified source data, a cleaned/curated zone for validated and reformatted data, and a consumption zone shaped for specific analytics or ML use cases.

4. Challenges

• Complexity — a lake is an assembled architecture, not a single tool, which raises the number of moving parts to operate.
• Lack of structure — storing anything in any format makes integrating data from diverse sources genuinely hard.
• Data quality — ensuring integrity across large volumes from diverse sources takes deliberate validation, not just storage.
• Cost — cost-effective for large volumes, but storage, processing and querying still add up without S3 tiering/lifecycle rules, Glue ETL to optimal formats, and partitioning.
• Integration — setting up pipelines across multiple sources and analytical tools (Athena, Redshift Spectrum, QuickSight, EMR, SageMaker) is itself a project.

5. Governance

Because data lands in raw form, governance has to be a deliberate strategy, not an afterthought:

6. Building with Lake Formation

AWS Lake Formation centralizes lake setup and permissions on top of S3 and the Glue Catalog:

• Blueprints & workflows — automate ingestion from common sources instead of hand-building Glue jobs for each.
• Centralized permissions — grant table- and column-level access once, enforced consistently across Athena, Redshift Spectrum, EMR and Glue.
• FindMatches — ML-based record matching/deduplication across datasets without hand-written matching rules.

Typical course-outline build sequence: 1) set up a simple data lake on raw S3 data, 2) build it properly with Lake Formation, 3) layer on security and access controls, 4) automate with blueprints/workflows, 5) visualize with QuickSight.

7. Schema Evolution

How safely a lake tolerates a changing schema depends entirely on file format, because Athena reads columns differently by format:

Keep the Glue Catalog current: schedule crawlers to run periodically, since Athena and other query engines misbehave once the catalog drifts from the actual data. Never mix structurally different files in the same S3 prefix — a crawler can create multiple tables from one prefix if the schemas underneath don't match, and query engines expect one consistent shape per path.

8. The Lakehouse & Medallion Architecture

A lakehouse adds warehouse-like guarantees — ACID transactions, schema enforcement, time travel — directly on top of lake storage, using an open table format: Delta Lake, Apache Iceberg, or Apache Hudi. It aims to close the gap in §2 without running two separate systems.

The medallion architecture organizes a lakehouse into three progressively refined zones:

Each zone is typically its own set of Delta/Iceberg tables, with a pipeline (often Spark or Glue ETL) promoting data from one zone to the next — the same bronze→silver→gold vocabulary maps directly onto the staging→core→mart layers described in the Data Warehouse reference doc.

9. Streaming Ingestion

Lakes increasingly ingest continuously rather than in nightly batches, landing data straight into the bronze zone as it arrives:

from pyspark.sql.functions import current_timestamp

spark.readStream.format("kinesis") \
    .option("streamName", "orders-stream") \
    .option("region", "us-east-1") \
    .option("startingPosition", "TRIM_HORIZON") \
    .load() \
  .withColumn("ingested_at", current_timestamp()) \
  .writeStream \
    .format("delta") \
    .option("checkpointLocation", "/checkpoints/bronze/orders") \
    .start("/lake/bronze/orders")

Kinesis Data Firehose can also write directly to S3 with no Spark job at all, trading transformation flexibility for operational simplicity — a good fit when bronze truly needs no logic beyond "land the raw record."

10. Proof of Concept First

Before committing significant time and resources, build a small proof-of-concept lake to validate the approach:

• Feasibility — confirm the proposed solution or technology actually works for this data.
• Risk — surface potential problems early, while they're still cheap to fix.
• Performance — test real storage, processing and query performance against representative data.
• Communication — a working POC aligns teams and stakeholders far faster than a design doc alone.

11. Building a Lakehouse on Databricks

Cloud storage (S3) ─► Unity Catalog (catalog · schemas · volumes · external locations)
  ─► Ingest (Auto Loader / Lakeflow Connect / Kafka) ─► BRONZE ─► SILVER ─► GOLD (Delta)
  ─► Orchestrate (Lakeflow Jobs / Declarative Pipelines)
  ─► Govern (grants, masks, lineage)
  ─► Serve (SQL warehouses, dashboards, BI, ML)

CREATE CATALOG IF NOT EXISTS fraud_dev MANAGED LOCATION 's3://bank-lakehouse-dev/managed';
CREATE SCHEMA IF NOT EXISTS fraud_dev.bronze;
CREATE SCHEMA IF NOT EXISTS fraud_dev.silver;
CREATE SCHEMA IF NOT EXISTS fraud_dev.gold;
CREATE VOLUME IF NOT EXISTS fraud_dev.bronze.landing;

GRANT USE CATALOG ON CATALOG fraud_dev TO `fraud_analysts`;
GRANT USE SCHEMA, SELECT ON SCHEMA fraud_dev.gold TO `fraud_analysts`;

12. Unity Catalog

Unity Catalog is Databricks' governance layer: one place that records what data exists, where it lives, who may access it, and how it is used. Delta stores the tables; Unity Catalog organises and secures them.

• Managed tables: Unity Catalog owns the files; dropping the table deletes the data. External tables: files stay in your own storage path.
• There is the full-featured Databricks Unity Catalog and an open-source Unity Catalog (2024, Linux Foundation) with fewer features.
• Comparable tools: Hive Metastore, AWS Glue Catalog + Lake Formation, Apache Polaris and Nessie (for Iceberg).

13. Data Governance Architecture

Federated (data mesh) governance

Domains own their data as products, a central council sets a small number of global rules, and the platform enforces those rules automatically. This is called federated computational governance.

• Global rules: classification, privacy, residency, canonical identifiers, interoperability, minimum quality and metadata, retention. Local decisions: which products to build, internal models, domain rules, release timing.
• Enforced as code: publishing fails without an owner and classification; tag-based masking; contract tests in CI/CD; region-locked infrastructure templates; quarantine on DQ failure.
• Data contracts specify schema, quality (completeness, freshness, duplicates), SLA, retention and the access process.
• Roll out with 2–3 motivated domains, measure (ownership %, DQ scores, access lead time), then scale.

BCBS 239 (risk data aggregation and reporting)

In practice: identify the critical data elements (CDEs) behind key risk reports; give each an owner, definition, lineage, DQ rules and a reconciliation to the general ledger; build a governed risk data platform with canonical models and MDM for counterparties; remove manual spreadsheet adjustments; define SLAs, including for stress scenarios; and keep evidence (scorecards, reconciliation logs, lineage) for supervisors.

Lineage and data quality at scale

• Lineage (business, technical, column-level) supports impact analysis, root-cause analysis, regulatory evidence and privacy requests. Capture it automatically with Unity Catalog, Purview, DataZone/Dataplex or OpenLineage (Spark, dbt, Airflow → Marquez), plus parsers for legacy code.
• DQ dimensions: completeness, validity, accuracy, consistency, uniqueness, timeliness, integrity.
• Check at every layer: ingestion (schema, volume, freshness), transformation (business rules), consumption (reconciliation, thresholds).
• Use a reusable rule library as code (Great Expectations, Deequ, dbt tests, Soda, Glue DQ) with severities: fail, quarantine or warn. Publish scorecards in the catalog and route issues to stewards. Start with CDEs.

Data residency and privacy across clouds

 Global control plane: unified catalog + classification tags · policy-as-code · central audit
        │ same policies pushed to each region
 INDIA data plane (primary + in-country DR)      EU data plane (own regions)
 raw PII and payment data stay; token vault;     EU PII stays; own keys;
 bank-controlled KMS/HSM keys                    local processing
        └── only aggregated / anonymised / tokenised data crosses borders ──┘

• Privacy lifecycle: consent and purpose, minimisation, automated retention, data-subject requests via lineage (crypto-shredding of token-vault entries), erasure vs AML retention decided by legal, breach response deadlines (CERT-In 6 hours, GDPR 72 hours).
• Common pitfalls: forgotten copies (backups, logs, test data, SaaS tools), policy drift between clouds, and assuming encryption alone solves residency.

How they connect: governance sets the rules, the platform enforces them, lineage and quality prove they work, and that evidence is what regulators ask for.

14. Banking Compliance & Regulation

Regulations change often, so verify current versions. This is a learning map, not legal advice.

Applied to a fraud platform: tokenise account, device and payee IDs; encrypt and restrict access; keep reasons and an audit trail for every alert with STR export; define retention; deploy in an Indian region with KMS and IAM; and document and monitor rules and models.