DataTerminology.comNEW
Also known as: Optimized Row Columnar
A columnar data storage format optimized for big data analytics with efficient compression and predicate pushdown.
DESCRIPTION
ORC (Optimized Row Columnar) is a columnar file format designed for efficient storage and fast reads in big data systems. It stores data in stripes with indexes and statistics to enable predicate pushdown and selective reads. ORC supports compression and encoding techniques for high compression ratios. It integrates with Hadoop ecosystems, including Hive, Spark, and Presto/Trino. Rich metadata and type awareness improve query performance and schema handling in data lakes. ORC competes with Parquet, with performance varying by workload and engine.
ALTERNATIVE DEFINITIONS
Wikipedia
Apache ORC (Optimized Row Columnar) is a free and open-source columnar data storage format for Hadoop ecosystems.
Emphasizes open-source nature and Hadoop focus.
Apache ORC
ORC is a self-describing, type-aware columnar file format designed for Hadoop workloads.
Highlights self-describing metadata and type awareness.
RELATED TERMS
Columnar Storage Apache Hive Apache Spark Predicate Pushdown Compression Data Lake Schema Evolution HDFS
COMPETING TERMS
RELATIONS
- Narrower: Apache ORC
KNOWLEDGE AREAS
Data Engineering Data Storage & Operations Data Architecture Cloud
TARGET AUDIENCE
Data Engineer Data Architect Cloud Architect Data Modeler
EXTERNAL SOURCES
- Wikipedia - Apache ORC
- Apache ORC - Project overview and resources
- AWS - Using ORC with Amazon Athena
TOP QUESTIONS
When should I choose ORC over Parquet?
ORC often excels with Hive and some engines for certain aggregations, thanks to stripe-level stats and encodings. Evaluate both on your workload and engine for best results.
Category: Alternatives
How does ORC improve query performance?
ORC uses columnar storage, stripes, indexes, and min/max stats for predicate pushdown. Engines skip irrelevant data to reduce I/O and speed scans.
Category: Concepts
Does ORC support schema evolution?
Yes. ORC stores schema metadata, enabling addition of fields and type handling. Engine-specific capabilities determine practical evolution support.
Category: Requirements
What compression does ORC support?
Common options include Zlib, Snappy, and Zstandard, selectable per file. Compression choice balances size and CPU cost.
Category: Overview
What are common mistakes with ORC in data lakes?
Writing too many small files and frequent schema changes hurt performance. Aim for proper file sizes and stable schemas for efficient scans.
Category: Mistakes
Is ORC suitable for streaming ingestion?
ORC targets batch writes. For streaming, buffer micro-batches and compact files periodically to avoid small-file problems.
Category: Best practices
How does ORC store metadata?
Each stripe keeps indexes and statistics; file-level metadata stores schema and column stats, enabling efficient skipping during reads.
Category: Concepts
Which engines natively read ORC?
Apache Hive, Spark, Presto/Trino, and many cloud analytics services support ORC natively or via connectors.
Category: Overview
What file size is recommended for ORC?
Target 128–512 MB per file depending on engine and cluster. Larger stripes improve scan efficiency but consider parallelism needs.
Category: Best practices