📦Avro

NEW

Also known as: Apache Avro

A schema-driven data serialization system with compact binary encoding and robust schema evolution.

DESCRIPTION

Apache Avro defines schemas in JSON and serializes data in a compact binary format. It enables schema evolution through explicit field definitions and compatibility rules. Avro integrates widely with Apache Kafka and schema registries for producer–consumer contracts. Language-agnostic bindings and code generation streamline interoperability. It is commonly used for event streaming, data exchange, and long-term storage where strict schemas and evolution are essential.

ALTERNATIVE DEFINITIONS

Apache Avro

Apache Avro™ is a data serialization system.

Concise and authoritative project definition.

Source

Wikipedia

Apache Avro is a row-oriented, language-neutral, data serialization system developed within the Apache Hadoop project.

Adds history and orientation within Apache Hadoop.

Source

RELATED TERMS

Apache Kafka Schema Registry JSON Schema Parquet Schema Evolution Serialization Binary Encoding Data Contracts

COMPETING TERMS

Protocol Buffers Apache Thrift

RELATIONS

• Broader: Binary Encoding Serialization
• Conforms to: JSON Schema

KNOWLEDGE AREAS

Data Engineering Real-time & Streaming Data Data Integration & Interoperability Data Storage & Operations

TARGET AUDIENCE

Data Engineer MLOps Engineer Software Developer Data Architect

EXTERNAL SOURCES

• Wikipedia - Apache Avro
• Apache Avro - Official Avro documentation
• Confluent - Using Avro with Schema Registry

TOP QUESTIONS

Why choose Avro for streaming data?

Avro’s compact binary encoding and explicit schemas reduce payload size and enforce contracts. With Schema Registry, it supports safe evolution across producers and consumers.

Category: Relevancy

How does Avro handle schema evolution?

Through compatibility rules (backward, forward, full) and default values for new fields. Registries validate compatibility during schema updates.

Category: Concepts

What is the difference between Avro and Protocol Buffers?

Avro stores schemas externally and is popular with Kafka; Protobuf uses .proto IDLs and code generation. Choice depends on ecosystem and tooling preferences.

Category: Alternatives

Is code generation required for Avro?

No. Avro supports generic records without code generation, though compiled classes can improve type safety and performance.

Category: Requirements

What are common Avro mistakes?

Changing field names or types without compatibility planning, and inconsistent namespaces. Always establish schema governance and versioning policies.

Category: Mistakes

How does Avro compare in size to JSON?

Avro is binary and typically much smaller than JSON, especially for repetitive field names and typed data, lowering bandwidth and storage costs.

Category: Overview

Can Avro be used for long-term data storage?

Yes. Store Avro with embedded schema references or data files containing schemas, ensuring future readability as schemas evolve.

Category: Best practices

How do I validate Avro schemas?

Use Avro libraries and Schema Registry to lint, register, and enforce compatibility. Automated checks in CI prevent breaking changes.

Category: How-to

Is Avro columnar like Parquet?

No. Avro is row-oriented; Parquet and ORC are columnar. Choose based on access patterns and analytics requirements.

Category: Misunderstanding