Apache Iceberg Reference

Metadata architecture, catalogs, hidden partitioning and evolution, time travel and branches, copy-on-write vs merge-on-read, maintenance, and Iceberg on AWS.

1. What Apache Iceberg Is

Apache Iceberg is an open table format, created at Netflix and now an Apache project, designed for huge analytic tables shared by many engines. It is defined by an open specification, so Spark, Flink, Trino, Athena, EMR, Snowflake, BigQuery, Dremio and DuckDB can all read and write the same tables.

2. Metadata Architecture

 CATALOG (Glue / REST / Hive / Nessie / Polaris)
   db.transactions ──► current metadata pointer
                           │
             s3://lake/transactions/metadata/00005-….metadata.json
             (schema, partition specs, sort orders, snapshots list, properties)
                           │  current snapshot
                           ▼
                 snap-…-manifest-list.avro      (one per snapshot)
                 lists manifests + partition ranges
                    │                 │
                    ▼                 ▼
             manifest-a.avro    manifest-b.avro   (track data / delete files
                    │                 │            with per-column stats)
                    ▼                 ▼
         data/…/00001.parquet   data/…/00002.parquet

• A commit writes new data files, manifests, a manifest list and a new metadata.json, then atomically swaps the catalog pointer from the old metadata file to the new one. If two writers race, the loser retries against the new metadata.
• Query planning reads metadata, not directory listings: manifest-list partition ranges and per-file column statistics prune files before any data is read. This scales to millions of files and works well on S3.
• Old snapshots keep their manifests and files until you expire them, which enables time travel.

3. Catalogs & Spark Setup

spark = (SparkSession.builder.appName("iceberg")
  .config("spark.jars.packages",
          "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1,"
          "org.apache.iceberg:iceberg-aws-bundle:1.6.1")
  .config("spark.sql.extensions",
          "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
  .config("spark.sql.catalog.glue", "org.apache.iceberg.spark.SparkCatalog")
  .config("spark.sql.catalog.glue.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog")
  .config("spark.sql.catalog.glue.warehouse", "s3://acme-lake/warehouse/")
  .config("spark.sql.catalog.glue.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
  .getOrCreate())

The catalog name (glue here) becomes the first part of table names: glue.db.transactions. Keep the runtime JAR matched to your Spark and Scala versions.

4. Creating Tables & Writing Data

CREATE TABLE glue.fraud.transactions (
  txn_id     STRING,
  account_id STRING,
  amount     DECIMAL(18,2),
  channel    STRING,
  event_ts   TIMESTAMP)
USING iceberg
PARTITIONED BY (days(event_ts), bucket(16, account_id))
TBLPROPERTIES ('format-version' = '2', 'write.merge.mode' = 'merge-on-read');

INSERT INTO glue.fraud.transactions SELECT * FROM staging_txn;

MERGE INTO glue.fraud.transactions t
USING updates s
ON t.txn_id = s.txn_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

# DataFrameWriterV2 API
df.writeTo("glue.fraud.transactions").append()
df.writeTo("glue.fraud.transactions").overwritePartitions()      # replace only partitions present in df
(df.writeTo("glue.fraud.daily_summary")
   .using("iceberg").partitionedBy(F.col("event_date"))
   .createOrReplace())

spark.table("glue.fraud.transactions").filter("event_ts >= '2026-10-25'")   # partition pruning via days(event_ts)

5. Hidden Partitioning & Partition Evolution

Iceberg stores the partition transform in metadata, so users never see or filter on a separate partition column. A query on event_ts >= '2026-10-25' prunes daily partitions automatically. With Hive-style tables, forgetting to also filter on event_date scans everything.

-- data volume grew: switch new data from daily to hourly partitions
ALTER TABLE glue.fraud.transactions REPLACE PARTITION FIELD days(event_ts) WITH hours(event_ts);
ALTER TABLE glue.fraud.transactions ADD PARTITION FIELD channel;
ALTER TABLE glue.fraud.transactions DROP PARTITION FIELD channel;

Partition evolution is a metadata-only change: existing files keep the old layout, new files use the new spec, and queries plan across both. No table rewrite is needed.

6. Schema Evolution

Iceberg tracks columns by unique field IDs, not by name or position, so schema changes are safe metadata operations. A renamed column keeps its ID and data, and a dropped-then-re-added column never resurrects old values.

ALTER TABLE glue.fraud.transactions ADD COLUMN device_id STRING AFTER channel;
ALTER TABLE glue.fraud.transactions RENAME COLUMN channel TO payment_channel;
ALTER TABLE glue.fraud.transactions ALTER COLUMN amount TYPE DECIMAL(20,2);   -- safe widening only
ALTER TABLE glue.fraud.transactions DROP COLUMN device_id;

Allowed type changes are widenings such as int → bigint, float → double, and increasing decimal precision.

7. Time Travel, Branches & Metadata Tables

SELECT * FROM glue.fraud.transactions VERSION AS OF 8749236512094857;   -- snapshot id
SELECT * FROM glue.fraud.transactions TIMESTAMP AS OF '2026-10-25 06:00:00';

-- inspect the table through metadata tables
SELECT * FROM glue.fraud.transactions.snapshots;    -- snapshot_id, operation, summary
SELECT * FROM glue.fraud.transactions.history;
SELECT * FROM glue.fraud.transactions.files;        -- data files with record counts and sizes
SELECT * FROM glue.fraud.transactions.partitions;

CALL glue.system.rollback_to_snapshot('fraud.transactions', 8749236512094857);

Write-Audit-Publish with branches

ALTER TABLE glue.fraud.transactions CREATE BRANCH audit;

-- 1. write today's load to the branch (main is untouched)
INSERT INTO glue.fraud.transactions.branch_audit SELECT * FROM staging_txn;

-- 2. run data-quality checks against the branch
SELECT COUNT(*) FROM glue.fraud.transactions VERSION AS OF 'audit' WHERE amount <= 0;

-- 3. publish: move main forward to the audited branch
CALL glue.system.fast_forward('fraud.transactions', 'main', 'audit');

Tags (ALTER TABLE … CREATE TAG eom_2026_10) pin a snapshot by name, for example month-end regulatory reports that must stay reproducible.

8. Copy-on-Write vs Merge-on-Read

• Format v2 delete files are position deletes (file + row position, written by Spark) or equality deletes (delete rows where key = value, typical from Flink upserts).
• Format v3 adds deletion vectors (compact bitmaps per data file), row lineage, and new types such as variant and nanosecond timestamps. Check engine support before upgrading.
• Merge-on-read tables need regular compaction, or read performance degrades as delete files pile up.

9. Table Maintenance

-- compact small files (and apply delete files)
CALL glue.system.rewrite_data_files(table => 'fraud.transactions');
CALL glue.system.rewrite_data_files(table => 'fraud.transactions',
                                    strategy => 'sort', sort_order => 'account_id, event_ts');

-- drop old snapshots (limits time travel, frees files)
CALL glue.system.expire_snapshots(table => 'fraud.transactions',
                                  older_than => TIMESTAMP '2026-10-01 00:00:00',
                                  retain_last => 20);

-- delete files no snapshot references (failed writes, crashed jobs)
CALL glue.system.remove_orphan_files(table => 'fraud.transactions');

-- consolidate many small manifests after lots of small commits
CALL glue.system.rewrite_manifests('fraud.transactions');

Schedule these (e.g. daily compaction, weekly snapshot expiry). Streaming writers create a snapshot per commit, so without expiry the metadata grows without limit.

10. Iceberg on AWS

-- Athena (engine v3): create, upsert, maintain
CREATE TABLE fraud.transactions (
  txn_id string, account_id string, amount decimal(18,2), event_ts timestamp)
PARTITIONED BY (day(event_ts))
LOCATION 's3://acme-lake/warehouse/fraud/transactions/'
TBLPROPERTIES ('table_type' = 'ICEBERG');

MERGE INTO fraud.transactions t USING fraud.staging s ON t.txn_id = s.txn_id
WHEN MATCHED THEN UPDATE SET amount = s.amount
WHEN NOT MATCHED THEN INSERT (txn_id, account_id, amount, event_ts)
                      VALUES (s.txn_id, s.account_id, s.amount, s.event_ts);

OPTIMIZE fraud.transactions REWRITE DATA USING BIN_PACK;
VACUUM fraud.transactions;

11. Common Interview Questions