📦Apache ORC

NEW

Also known as: Optimized Row Columnar

A self-describing, columnar data file format optimized for analytics in the Hadoop ecosystem.

DESCRIPTION

Apache ORC stores columnar data with built-in statistics, compression, and indexing for fast queries. It supports predicate pushdown and efficient I/O, making it suitable for large-scale analytics. ORC integrates closely with Hive and works with engines like Spark and Presto. File-level and stripe-level metadata accelerates selective reads. It reduces storage via compression schemes like ZLIB and SNAPPY. While Parquet is broadly used across platforms, ORC excels in Hive-centric environments. Both formats aim to optimize analytical workloads on data lakes.

ALTERNATIVE DEFINITIONS

Apache ORC

The Optimized Row Columnar (ORC) file format provides a highly efficient way to store data.

Highlights efficiency and purpose as a storage format.

Source

Wikipedia

Apache ORC (Optimized Row Columnar) is a free and open-source column-oriented data storage format.

Positions ORC as an open-source, column-oriented format.

Source

RELATED TERMS

Avro Hadoop Apache Hive Columnar Storage Compression Predicate Pushdown Data Lake Apache Spark Apache Iceberg

COMPETING TERMS

Parquet

RELATIONS

• Broader: Columnar Storage ORC

KNOWLEDGE AREAS

Data Engineering Data Storage & Operations Real-time & Streaming Data

TARGET AUDIENCE

Data Engineer Data Architect Cloud Architect Analytics Engineer

EXTERNAL SOURCES

• Wikipedia - Apache ORC
• Apache ORC - Official ORC documentation
• Apache Hive - Hive usage and ORC options
• Cloudera - ORC format in enterprise Hadoop

TOP QUESTIONS

How does Apache ORC improve query performance?

ORC stores column statistics and indexes, enabling predicate pushdown and skipping irrelevant blocks. Compression reduces I/O, accelerating analytical queries.

Category: Performance

When should I choose Apache ORC over Parquet?

Prefer ORC in Hive-centric ecosystems or when ORC-specific optimizations are available. If multi-engine interoperability is key, Parquet may offer broader support.

Category: Comparison

What compression options exist in Apache ORC?

Common codecs include ZLIB, SNAPPY, and LZO. Choose based on speed versus compression ratio and cluster capabilities.

Category: Configuration

Does Apache ORC support schema evolution?

Yes, ORC supports adding columns and certain type promotions. Coordinate with table metadata (e.g., Hive Metastore) for consistent interpretation.

Category: Governance

What engines natively read Apache ORC?

Apache Hive, Spark, Presto/Trino, and Impala can read ORC. Many cloud platforms provide native connectors as well.

Category: Tooling

How does predicate pushdown work in Apache ORC?

ORC stores min/max and bloom filters per stripe, allowing readers to skip non-matching stripes, reducing I/O significantly.

Category: Concept

What is a stripe in Apache ORC?

A stripe is a large block within an ORC file containing data, indexes, and statistics for column chunks, enabling efficient reads and skips.

Category: Concept

Is Apache ORC suitable for small files?

ORC shines with large files. Many small files harm performance; compact small files into larger ones for optimal throughput.

Category: BestPractice

How does Apache ORC handle nulls and types?

ORC encodes column types with null bitmaps and type-specific encodings, preserving schema fidelity and reducing storage through compression.

Category: DataModeling