🧱Parquet

NEW

Also known as: Apache Parquet

An open, column-oriented data file format designed for efficient analytics with compression and predicate pushdown.

DESCRIPTION

Apache Parquet stores data column-wise, providing high compression and efficient scans for analytical workloads. It embeds schema and statistics to enable predicate pushdown and column pruning. Parquet integrates with Spark, Presto/Trino, Hive, and most cloud warehouses and lakes. It supports nested data structures and is a de facto standard for big data analytics. Compression codecs (SNAPPY, ZSTD, GZIP) balance speed and size. Parquet’s broad ecosystem support makes it ideal for interoperable data lake architectures. It complements table formats like Apache Iceberg and Delta Lake for ACID and governance.

ALTERNATIVE DEFINITIONS

Apache Parquet

Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval.

Highlights column orientation and efficiency goals.

Source

Wikipedia

Apache Parquet is a free and open-source column-oriented data storage format of the Apache Hadoop ecosystem.

Places Parquet within the Hadoop ecosystem and open-source context.

Source

RELATED TERMS

Avro Columnar Storage Predicate Pushdown Data Lake Apache Spark Presto Apache Iceberg Delta Lake Compression

COMPETING TERMS

Apache ORC

KNOWLEDGE AREAS

Data Engineering Data Storage & Operations Real-time & Streaming Data

TARGET AUDIENCE

Data Engineer Data Architect Cloud Architect Analytics Engineer Data Scientist

EXTERNAL SOURCES

• Wikipedia - Apache Parquet
• Apache Parquet - Official Parquet documentation
• Apache Spark - Using Parquet in Spark SQL
• Databricks - Parquet in lakehouse environments

TOP QUESTIONS

Why is Parquet preferred for analytics?

Columnar storage enables column pruning and compression, reducing I/O dramatically. Embedded statistics enable predicate pushdown for faster queries.

Category: Performance

How does Parquet handle nested data?

Parquet uses repetition and definition levels to encode nested structures efficiently, preserving complex schemas for analytics.

Category: DataModeling

Which compression codecs are recommended for Parquet?

SNAPPY for speed, ZSTD for balanced compression, and GZIP for higher ratios. Benchmark with your data to choose appropriately.

Category: Configuration

How does Parquet compare to ORC?

Both are columnar and efficient. Parquet often has broader engine support; ORC integrates tightly with Hive. Performance varies by workload and engine.

Category: Comparison

Is Parquet suitable for small files?

Parquet performs best with larger files. Compact small files to reduce metadata overhead and optimize scanning.

Category: BestPractice

Can Parquet ensure schema evolution?

Yes, it supports adding columns and compatible type changes. Coordinate changes with table metadata and consumers.

Category: Governance

What engines support Parquet natively?

Spark, Trino/Presto, Hive, Flink, and most cloud data warehouses and lake platforms support Parquet out of the box.

Category: Tooling

How does Parquet store column statistics?

It keeps min/max and count per column chunk, enabling readers to skip non-relevant data during scans.

Category: Concept

When should Parquet not be used?

Avoid for high-frequency record updates or OLTP-style workloads. Row-oriented stores or databases are better choices there.

Category: Application