🧊Apache Iceberg

NEW

Also known as: Iceberg

An open table format for large analytic datasets that adds ACID, schema evolution, and time travel to data lakes.

DESCRIPTION

Apache Iceberg defines a high-performance table format for data lakes with transactional guarantees. It supports schema evolution, partition evolution, hidden partitioning, snapshots, and time travel. Iceberg decouples metadata from physical storage and enables efficient planning and incremental processing. It integrates with engines like Spark, Flink, Trino, and Presto. Competing formats include Delta Lake and Apache Hudi. Iceberg is widely adopted for lakehouse architectures.

ALTERNATIVE DEFINITIONS

Apache Software Foundation

Apache Iceberg is an open table format for huge analytic datasets.

Concise mission statement highlighting scope and openness.

Source

AWS Glue Documentation

Apache Iceberg is a high-performance format for huge analytic tables that brings the reliability and simplicity of SQL tables to big data, while making it possible for engines to safely work with the same tables.

Emphasizes SQL-like reliability and multi-engine safety.

Source

Trino

The Iceberg connector allows querying and managing data stored in Apache Iceberg tables, a table format for large-scale analytics datasets.

Notes interoperability via Trino connector and analytics focus.

Source

RELATED TERMS

Tabular Data Data Lakehouse Parquet ACID Time Travel Partitioning Metadata Apache Spark

COMPETING TERMS

Delta Lake Apache Hudi

KNOWLEDGE AREAS

Data Engineering Data Storage & Operations Data Architecture Cloud Platform & Tooling

TARGET AUDIENCE

Data Engineer Data Architect Cloud Architect MLOps Engineer

EXTERNAL SOURCES

• Wikipedia - Apache Iceberg
• Apache Iceberg - Official Iceberg website
• Apache Spark - Spark integration with Iceberg
• AWS - AWS Glue guide to Iceberg
• Dremio - Iceberg tables in Dremio

TOP QUESTIONS

Why choose Apache Iceberg for a data lake?

It provides ACID transactions, schema evolution, and time travel, enabling reliable lakehouse analytics across multiple query engines.

Category: Relevancy

How does Iceberg compare to Delta Lake?

Both add transactions to lakes; Iceberg focuses on open standardization and broad engine support, while Delta is tightly integrated with Databricks.

Category: Comparison

What engines integrate with Apache Iceberg?

Spark, Flink, Trino, Presto, Hive, and others support reading and writing Iceberg tables.

Category: Tools

How does Iceberg enable time travel?

Through immutable snapshots and manifests that track table state, allowing queries at specific points in time.

Category: Concept

What is hidden partitioning in Iceberg?

Partitioning logic is stored in metadata, avoiding user errors and automatic predicate pushdown without exposing partition columns.

Category: Concept

What are common mistakes when adopting Iceberg?

Neglecting metadata maintenance (vacuum), misconfigured write isolation, and inconsistent table property management can degrade performance.

Category: Mistakes

What storage systems support Iceberg?

Object stores like S3, ADLS, and GCS, as well as HDFS and local filesystems.

Category: Tools

How does Iceberg handle schema evolution?

It assigns stable field IDs and supports column adds, renames, and type widening without rewriting entire tables.

Category: Mechanism

What is the governance benefit of Iceberg?

Transactional integrity and lineage-friendly metadata improve auditability and reproducibility across teams and tools.

Category: Governance