➕Append-Only Log

NEW

A sequential record where new entries are appended, preserving full history without in-place updates.

DESCRIPTION

Append-only logs record events as a growing sequence, enabling durable history, replay, and efficient replication. They underpin systems like Kafka, write-ahead logging, and event sourcing. Compaction or snapshotting manages growth while retaining semantics. Append patterns improve write performance and crash recovery. Consumers can process from offsets, supporting streaming analytics and CDC. Immutability of past entries simplifies concurrency control. Operationally, segmenting, retention, and checkpoints balance performance and storage.

ALTERNATIVE DEFINITIONS

Apache Kafka

The log is an append-only sequence of records.

Emphasizes the core append-only property in Kafka’s design.

Source

Martin Fowler

Event sourcing involves storing all changes to application state as a sequence of events.

Connects append-only logs to event-sourced system design.

Source

RELATED TERMS

Immutable Log Write-Ahead Logging Transaction Log Event Sourcing Change Data Capture Log-Structured Storage Apache Kafka Log-Structured Merge-Tree Replication Checkpointing

COMPETING TERMS

Mutable Storage

RELATIONS

• Broader: Append-Only Immutable Log Transaction Log

KNOWLEDGE AREAS

Data Engineering Real-time & Streaming Data Data Storage & Operations Software Engineering

TARGET AUDIENCE

Data Engineer Software Developer DevOps Engineer Data Architect

EXTERNAL SOURCES

• Wikipedia - Write-ahead logging
• Apache Kafka - Kafka log design and usage
• Wikipedia - Log-structured storage concept
• PostgreSQL - WAL append logging intro

TOP QUESTIONS

Why use an Append-Only Log?

It provides durable history, simplifies concurrency, and supports replay for recovery and analytics. It is foundational to streaming systems.

Category: Relevancy

How do systems manage log growth?

They use retention, compaction, and snapshotting to prune or condense history while preserving semantics needed for recovery and consumers.

Category: Management

What’s the difference between Append-Only Log and Immutable Log?

Append-only emphasizes write pattern; immutable emphasizes tamper resistance. Many systems are both append-only and immutable.

Category: Comparison

How do consumers read from an Append-Only Log?

They use offsets or sequence numbers to process records in order, enabling exactly-once or at-least-once semantics depending on design.

Category: How-To

How does an Append-Only Log aid recovery?

Replaying records rebuilds state after crashes, as in write-ahead logging. Checkpoints accelerate recovery by starting from snapshots.

Category: Example

What are common pitfalls with Append-Only Logs?

Unbounded growth, insufficient compaction, and poor partitioning lead to storage pressure and hot spots.

Category: Mistakes

Is an Append-Only Log suitable for OLTP data?

Directly storing OLTP records in append-only form is uncommon; instead, logs complement OLTP via WAL, CDC, or event sourcing.

Category: Opinion

How does compaction work in logs like Kafka?

Compaction retains the latest value per key, discarding older versions while keeping the append-only write model.

Category: Definition

What hardware considerations help Append-Only Logs?

Fast sequential writes and NVMe storage benefit append-only workloads. Separate IO paths for reads and compaction improve throughput.

Category: Best practices