🗜️Data Compression

NEW

Also known as: Data Encoding

Encoding data to reduce size for storage or transmission using lossless or lossy techniques.

DESCRIPTION

Data compression reduces bytes via algorithms like Huffman, LZ77, DEFLATE, Snappy, and Zstandard. Lossless methods preserve exact content; lossy methods trade fidelity for higher ratios in media. Compression impacts performance: fewer bytes over networks, but added CPU cost. Analytics formats like Parquet leverage columnar compression and encoding. Choosing codecs involves data type, speed, ratio, and compatibility.

ALTERNATIVE DEFINITIONS

Wikipedia

In computer science and information theory, data compression is the process of encoding information using fewer bits than the original representation.

Situates compression within information theory using fewer bits.

Source

IETF RFC 1951

This specification defines the DEFLATE compressed data format, which uses a combination of the LZ77 algorithm and Huffman coding.

Defines a widely used lossless compression format and its algorithmic basis.

Source

Google Cloud

BigQuery can load compressed files; supported compression types include GZIP, BZIP2, and DEFLATE.

Operational perspective on supported compression types for analytics loading.

Source

RELATED TERMS

Lossless Compression Lossy Compression Entropy Huffman Coding Lempel–Ziv DEFLATE Zstandard Snappy Parquet Columnar Storage

COMPETING TERMS

Uncompressed Storage Data Expansion

RELATIONS

• Broader: Compression

KNOWLEDGE AREAS

Data Storage & Operations Infrastructure Data Engineering Mathematics

TARGET AUDIENCE

Data Engineer Software Developer Cloud Architect Data Architect

EXTERNAL SOURCES

• Wikipedia - Data compression
• IETF - DEFLATE compression specification
• Zstandard - Zstd high-performance codec docs
• Apache Parquet - Columnar format and compression
• Microsoft - Compression impact on performance

TOP QUESTIONS

What’s the difference between lossless and lossy Data Compression?

Lossless preserves exact data; lossy discards information for higher ratios, suitable for audio/video.

Category: Definition

How does Data Compression affect performance?

It reduces transfer time but adds CPU overhead. Net benefit depends on data characteristics and codec speed.

Category: Performance

Which codecs are common in analytics?

Snappy and Zstandard are popular for Parquet due to speed and reasonable ratios.

Category: Overview

How to choose a compression codec?

Consider data type, ratio needs, decode speed, ecosystem support, and streaming vs batch requirements.

Category: Best practices

Can Data Compression hurt latency?

Yes, if CPU overhead outweighs byte savings. Benchmark under representative load.

Category: Risks

What’s a common mistake with Data Compression?

Compressing already compressed data, or using slow codecs in low-latency paths.

Category: Mistakes

Does columnar storage improve compression?

Yes. Homogeneous columns compress more effectively using dictionary and run-length encodings.

Category: Example

How to monitor compression effectiveness?

Track compression ratios, CPU utilization, read/write times, and storage costs.

Category: Observability

Is encryption compatible with Data Compression?

Compress before encrypting; encrypted data typically resists compression due to randomness.

Category: Best practices