DataTerminology.comNEW
A data redundancy technique that splits data into fragments with parity information to reconstruct lost pieces after failures.
DESCRIPTION
Erasure coding improves durability by encoding data into shards plus parity and distributing them across nodes or disks. Compared to replication, it offers similar fault tolerance with significantly lower storage overhead. It is widely used in distributed storage systems (Ceph, HDFS, object stores) and cold storage. Implementations choose coding schemes and k+m parameters to balance durability and performance. Erasure coding affects write/read latency and recovery costs, requiring careful design for workload patterns.
ALTERNATIVE DEFINITIONS
Wikipedia
In coding theory, an erasure code is a forward error correction code designed to protect data against erasures.
Places erasure coding within coding theory and emphasizes protection against erasures.
Apache Hadoop
Erasure coding (EC) in HDFS is a method for storing data that provides fault tolerance with less storage overhead than replication.
Highlights HDFS implementation and storage overhead savings versus replication.
Ceph Documentation
Erasure coded pools store data using erasure coding to reduce storage overhead compared to replicated pools while providing reliability.
Explains Ceph pools’ reliability with reduced overhead relative to replication.
Backblaze
Erasure coding is a data protection method that breaks data into small pieces, creates parity information, and spreads everything across drives.
Vendor-oriented explanation focusing on practical storage layout.
RELATED TERMS
Error-Correcting Code Parity RAID Reed–Solomon Code Persistence Distributed Storage Ceph HDFS
COMPETING TERMS
KNOWLEDGE AREAS
Infrastructure Data Storage & Operations Data Engineering Cloud
TARGET AUDIENCE
Cloud Architect DevOps Engineer Data Engineer Security Architect Solution Architect
EXTERNAL SOURCES
- Wikipedia - Erasure code
- Apache Hadoop - HDFS erasure coding design and usage
- Ceph - Ceph erasure-coded pool operations
- Google Cloud - GCS storage classes durability design
- AWS - S3 storage classes and durability
TOP QUESTIONS
Why choose Erasure Coding over replication?
It delivers comparable durability with much lower storage overhead, reducing costs in large-scale storage systems.
Category: Relevancy
What are common Erasure Coding parameters?
Schemes use k data shards and m parity shards (e.g., 6+3) balancing durability and overhead.
Category: Requirements
What are drawbacks of Erasure Coding?
Higher CPU usage and potentially slower writes and recovery compared to replication; careful tuning is required.
Category: Risks
Where is Erasure Coding used in practice?
Distributed filesystems, object storage, cold archives, and multi-AZ cloud storage architectures.
Category: Example
How does Erasure Coding impact availability?
It sustains failures by reconstructing data from parity, but recovery may increase latency.
Category: Opinion
What is a competing approach to Erasure Coding?
Replication stores full copies for simplicity and faster recovery, at higher storage cost.
Category: Alternatives
How is Erasure Coding implemented in HDFS?
Data blocks are grouped and parities computed, reducing overhead versus triple replication.
Category: Example
What coding schemes are common?
Reed–Solomon codes are widely used; newer LDPC and fountain codes appear in specific contexts.
Category: Overview
How do I tune Erasure Coding for workloads?
Choose k+m, placement policy, and stripe size based on read patterns, failure domains, and recovery performance.
Category: Management