DataTerminology.comNEW
Reducing data size by exploiting redundancy and irrelevance, using lossless or lossy methods to save storage and bandwidth.
DESCRIPTION
Compression encodes information more compactly to reduce storage and transmission costs. Lossless techniques preserve exact data, while lossy methods discard imperceptible or low-value information for higher ratios. Modern algorithms combine dictionary, predictive, and entropy coding stages. Choices involve trade-offs among speed, ratio, and resource usage. In analytics and AI, columnar and model-aware compression improve throughput and cache locality. Compression impacts latency, power, and costs in data platforms. Monitoring compression ratio and CPU overhead ensures optimal configurations.
ALTERNATIVE DEFINITIONS
Cloudflare Learning
Data compression is the process of encoding information using fewer bits than the original representation.
General, succinct definition tied to web performance.
Encyclopaedia Britannica
Data compression, process of reducing the amount of data needed to store or transmit a piece of information.
Emphasizes storage and transmission economy.
Wikipedia
In signal processing and information theory, data compression is the process of encoding information using fewer bits.
Situates compression within information theory context.
RELATED TERMS
Lossless Compression Lossy Compression Source Coding Huffman Coding Lempel–Ziv Zstandard JPEG MPEG Delta Encoding Shannon Entropy
COMPETING TERMS
Entity Matching Encoding
RELATIONS
KNOWLEDGE AREAS
Data Storage & Operations Data Engineering Infrastructure Data Science AI
TARGET AUDIENCE
Data Engineer Cloud Architect Software Developer Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Data compression
- Zstandard - Zstd compression algorithm documentation
- IETF - GZIP file format specification
- Apache ORC - Columnar compression in ORC
TOP QUESTIONS
What is data compression and why is it used?
It encodes data using fewer bits to save storage and bandwidth. It also improves I/O throughput, often at the cost of CPU.
Category: Relevancy
What’s the difference between lossless and lossy compression?
Lossless preserves exact data; lossy discards less important information for higher ratios, common in media formats like JPEG and MP3.
Category: Comparison
How do I choose a compression algorithm?
Balance ratio, speed, and resource use. Zstd and LZ4 target speed; gzip is ubiquitous; Brotli excels for web text; Parquet uses columnar encodings.
Category: Advice
What are common mistakes with compression?
Double-compressing data, compressing already compressed files, or using high CPU settings on hot paths. Measure end-to-end performance.
Category: Mistakes
How does compression affect latency?
It reduces I/O time but adds CPU overhead. Net latency depends on data size, CPU speed, and algorithm choices.
Category: Performance
Is compression useful for structured analytics data?
Yes. Columnar formats exploit repetition and types with encodings like RLE and dictionary, boosting scan performance.
Category: Benefits
What is entropy in compression?
Entropy measures information content and sets a lower bound on average code length. Lower entropy enables greater compression.
Category: Concepts
How to monitor compression effectiveness?
Track compression ratio, CPU time, throughput, and cache hit rates. Use representative workloads for benchmarking.
Category: Management
Can compression harm data quality?
Lossy compression can; lossless cannot. Choose lossless for data pipelines requiring exactness.
Category: Risks