What Is Apache Iceberg? A Practical Introduction

Apache Iceberg is an open table format for large analytic datasets. It gives data files in a data lake the structure and management features people expect from a database table, while allowing different compute engines to work with the same data.

This is useful when a team wants to keep data in low-cost object storage but still needs reliable reads and writes, predictable schema changes, and a clear view of how a table has changed over time.

At the top level, Iceberg’s metadata connects a table’s current snapshot to the manifests that describe its data files:

flowchart TD
    Metadata["Table metadata<br/>metadata.json<br/>schema, partition specs, snapshots"]
    Snapshot["Current snapshot"]
    ManifestList["Manifest list"]
    Manifests["Manifest files<br/>data and delete file entries"]
    DataFiles["Data and delete files<br/>for example, Parquet"]

    Metadata --> Snapshot
    Snapshot --> ManifestList
    ManifestList --> Manifests
    Manifests --> DataFiles

The problem with a folder of files

A collection of Parquet files in a storage bucket is not, by itself, a well-managed table. Engines need a reliable way to know which files belong to the table, what their schemas are, and which version of the data they should read. If writers update files while readers are querying them, it also matters that readers see a consistent table state.

Without a shared table format, teams often end up relying on engine-specific conventions, manual partition management, or custom processes to coordinate those details.

What Iceberg adds

Iceberg stores metadata alongside the data files. That metadata tracks the table’s current state and the files that make up each version. A write creates a new snapshot of the table; readers use a consistent snapshot instead of discovering files by listing a directory.

This metadata model provides a foundation for capabilities such as:

• Reliable table updates: readers can continue using a consistent snapshot while a new table state is committed.
• Schema evolution: columns can be added, renamed, or otherwise changed without treating every data file as an identical, fixed schema.
• Partition evolution: the table’s partitioning can change over time without requiring every query or reader to know the physical layout.
• Historical reads: snapshots can be used to inspect or query earlier table states, subject to the table’s retention and maintenance settings.

These features are managed at the table level, rather than being tied to a single query engine or to the names of files in a directory.

Iceberg is not a query engine

Iceberg is a table format, not a database server, storage system, or processing engine. Your data still lives in files, and compute engines do the work of reading, transforming, and writing it. Iceberg defines how table state and metadata are represented so compatible engines can coordinate around the same table.

That separation lets a lakehouse use different tools for ingestion, transformation, and analysis while keeping the table’s organization and history consistent.

When should you consider it?

Apache Iceberg is worth considering when you need to manage large analytical tables on object storage, especially when:

• more than one engine needs to read or write the same tables;
• datasets change incrementally rather than being replaced as a whole;
• schemas or partition strategies need to evolve; or
• you need reproducible reads of earlier table states.

It is not automatically the right choice for every dataset. The value depends on your engines’ Iceberg support, your write patterns, and the operational work required to maintain table metadata and old snapshots.

A useful mental model

Think of Iceberg as a layer of table metadata that sits between your data files and the engines that use them:

Query and processing engines
             |
       Iceberg table
   (metadata and snapshots)
             |
       Data files
     in object storage

The data remains in files, but Iceberg makes the table’s state explicit and trackable. That is the core idea behind using it in a lakehouse.