Apache Hudi Reference

Copy-on-Write vs Merge-on-Read, upserts and incremental queries, the timeline, indexing, table services, HoodieStreamer, and a Delta vs Iceberg vs Hudi comparison.

1. What Apache Hudi Is

Apache Hudi (Hadoop Upserts Deletes and Incrementals) is an open table format and data-lake platform created at Uber. It was built specifically for record-level upserts and deletes on lake storage and for incremental processing: pulling only the records that changed since a given commit. That makes it a strong fit for CDC ingestion and near-real-time lakes.

2. Copy-on-Write vs Merge-on-Read

3. Spark Setup & Upserts

spark = (SparkSession.builder.appName("hudi")
  .config("spark.jars.packages", "org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.0")
  .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
  .config("spark.sql.extensions", "org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
  .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
  .getOrCreate())

hudi_options = {
    "hoodie.table.name": "transactions",
    "hoodie.datasource.write.table.type": "MERGE_ON_READ",       # or COPY_ON_WRITE
    "hoodie.datasource.write.recordkey.field": "txn_id",          # unique key (composite: "a,b")
    "hoodie.datasource.write.partitionpath.field": "event_date",
    "hoodie.datasource.write.precombine.field": "updated_at",     # latest wins when keys collide
    "hoodie.datasource.write.operation": "upsert",
    "hoodie.datasource.write.hive_style_partitioning": "true",    # event_date=2026-10-25/
}

(updates_df.write.format("hudi")
    .options(**hudi_options)
    .mode("append")                     # append = upsert into existing table
    .save("s3://acme-lake/hudi/transactions"))

The precombine field decides which record wins when the same key appears more than once, within a batch or against existing data. Usually it is an update timestamp or a CDC sequence number.

4. Reading: Snapshot, Incremental & Time Travel

path = "s3://acme-lake/hudi/transactions"

snapshot = spark.read.format("hudi").load(path)

read_optimized = (spark.read.format("hudi")
    .option("hoodie.datasource.query.type", "read_optimized")
    .load(path))

incremental = (spark.read.format("hudi")
    .option("hoodie.datasource.query.type", "incremental")
    .option("hoodie.datasource.read.begin.instanttime", "20261025000000")   # exclusive
    .load(path))

as_of = (spark.read.format("hudi")
    .option("as.of.instant", "20261025120000")
    .load(path))

Every record carries metadata columns: _hoodie_commit_time, _hoodie_commit_seqno, _hoodie_record_key, _hoodie_partition_path and _hoodie_file_name. A downstream job stores the last _hoodie_commit_time it processed and uses it as the next incremental begin time.

5. The Timeline

The .hoodie/ folder holds the timeline: every action on the table as an instant with a timestamp, an action type and a state.

Each instant moves through requested → inflight → completed. Readers only see completed instants, which is how Hudi provides atomic commits; a failed write is rolled back automatically on the next write.

6. Indexing

An upsert must find which file group already holds each key. The index makes that lookup fast instead of scanning the whole table.

Trap: with a non-global index, uniqueness is only per partition. If a record's partition field changes (e.g. its date is corrected), you get a duplicate in the new partition unless you use a global index with partition-path updates enabled.

7. Table Services

Services can run inline (after a write, simple but adds latency), asynchronously alongside a streaming writer, or as separate scheduled jobs.

8. HoodieStreamer & Hudi on AWS

HoodieStreamer (formerly DeltaStreamer) is a ready-made Spark ingestion job: it reads from Kafka, files or JDBC, applies an optional transformation, upserts into Hudi, and checkpoints its position.

spark-submit \
  --class org.apache.hudi.utilities.streamer.HoodieStreamer \
  hudi-utilities-slim-bundle_2.12-1.0.0.jar \
  --table-type MERGE_ON_READ \
  --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
  --source-ordering-field updated_at \
  --target-base-path s3://acme-lake/hudi/transactions \
  --target-table transactions \
  --props file:///opt/config/kafka-source.properties \
  --continuous                       # keep running; omit for a single batch

9. Delta vs Iceberg vs Hudi

The formats are converging: Delta UniForm exposes Iceberg metadata, Apache XTable translates metadata between all three, and REST catalogs serve tables to many engines. Choose by your primary engines, catalog strategy and write pattern (batch vs CDC/upserts).

10. Common Interview Questions