➕Append-Only

NEW

Also known as: Immutable Append

A data pattern where records are only appended, never updated or deleted in place.

DESCRIPTION

Append-only systems record new facts by adding entries without modifying or deleting existing data. This simplifies concurrency, recovery, and auditability, and aligns with event-driven architectures. Querying current state may require compaction or aggregation. Many streaming logs and data lakes adopt append-only ingestion, with downstream views materializing snapshots. Trade-offs include growing storage and higher read-time computation unless managed with compaction or indexing. Governance benefits from immutable history and clear lineage.

ALTERNATIVE DEFINITIONS

Apache Kafka

The log is an append-only sequence of records ordered by time.

Canonical description of append-only logs in a leading streaming platform.

Source

Martin Fowler

With Event Sourcing, all changes to application state are stored as a sequence of events.

Highlights the immutable sequence concept foundational to append-only designs.

Source

Delta Lake

Use the delta.appendOnly table property to prevent operations that rewrite, delete, or update data in a table.

Introduces a configuration to enforce append-only behavior in lakehouse tables.

Source

RELATED TERMS

Immutable Log Event Sourcing Transaction Log Change Data Capture Time Travel Log Compaction Idempotency Delta Lake Kafka Topic Soft Delete

COMPETING TERMS

Mutable Tables Update-in-place

RELATIONS

• Broader: Immutable Log
• Narrower: Append-Only Log

KNOWLEDGE AREAS

Data Engineering Real-time & Streaming Data Data Architecture Data Storage & Operations

TARGET AUDIENCE

Data Engineer Software Developer MLOps Engineer Data Architect Analytics Engineer

EXTERNAL SOURCES

• Wikipedia - Event sourcing
• Apache Kafka - Append-only log concepts and APIs
• Delta Lake - Configuration and table properties
• Martin Fowler - Event sourcing principle and benefits

TOP QUESTIONS

Why use an Append-only pattern?

It improves auditability, simplifies concurrency and recovery, and aligns with event-driven designs by treating data as immutable facts.

Category: Relevancy

How do you query current state with Append-only data?

Materialize views via compaction, snapshotting, or aggregations that fold events into the latest state.

Category: Practice

What are risks of Append-only storage?

Storage growth and slower reads can occur without compaction, partitioning, and pruning strategies.

Category: Risk

How does Append-only compare to update-in-place?

Append-only favors immutable history and easier recovery; update-in-place offers simpler reads but complicates lineage and audit.

Category: Comparison

What is a common mistake with Append-only logs?

Lacking idempotent producers leads to duplicates; ensure deduplication keys and exactly-once semantics where needed.

Category: Mistakes

How do lakehouse tables enforce Append-only?

Use table properties (e.g., delta.appendOnly) and governance policies that block updates and deletes.

Category: Tools

Is Append-only suitable for GDPR requirements?

It can coexist with privacy via encryption, tokenization, tombstoning, and selective purge processes for right-to-erasure.

Category: Compliance

What workloads suit Append-only best?

Event logs, audit trails, CDC streams, and telemetry where history matters and ordering is important.

Category: Example

What is an alternative to Append-only?

Mutable tables with upserts or soft deletes, depending on audit, performance, and compliance needs.

Category: Alternatives