Open Table Formats Explained: Iceberg vs. Delta vs. Hudi

Open table formats act as metadata layers that transform collections of Parquet or ORC files in object storage into governed tables with database-like features such as ACID transactions, schema evolution, and time travel. Apache Iceberg, Delta Lake, and Apache Hudi each manage metadata to track file versions, enabling consistent reads during concurrent writes and the ability to query historical snapshots. These formats address limitations of traditional data lakes, where concurrent writes could corrupt data and updates required rewriting entire partitions without reliable consistency guarantees.

Apache Iceberg uses a metadata tree with manifest files and lists to efficiently track data files, supporting schema and partition evolution without rewriting underlying data. Delta Lake, developed by Databricks, records every write in an ordered transaction log with checkpoint files, ensuring consistent views during concurrent operations. Apache Hudi prioritizes fast record-level updates through indexes that locate specific files, making it suitable for change-data-capture pipelines and near-real-time ingestion workflows.

Iceberg’s layered metadata structure enables time travel by maintaining immutable snapshots of table states, allowing queries against previous versions for auditing or rollback. Delta Lake’s transaction log and catalog commits, now generally available, centralize governance and enable multi-table transactions, aligning with Iceberg’s catalog-oriented approach. Hudi’s design focuses on efficient updates and incremental processing, leveraging row lineage and deletion vectors to minimize data rewrites in streaming and CDC scenarios.

Parquet and ORC serve as columnar file formats, while open table formats like Iceberg, Delta Lake, and Hudi provide the table abstraction on top of them. These formats extend data skipping by caching Parquet-level statistics at the metadata layer, improving query performance on large datasets. Time travel and incremental processing are core capabilities, with row lineage and deletion vectors further optimizing change tracking and reducing storage overhead in evolving datasets.