🧩Checkpointing

NEW

Also known as: Data checkpoint

Periodically persists consistent operator state so streaming jobs can recover correctly after failures.

DESCRIPTION

Checkpointing captures a consistent, incremental snapshot of a streaming application’s state at regular intervals. Upon failure, the system restarts from the latest checkpoint to maintain processing guarantees. Checkpoints are typically automated, lightweight, and discarded over time as new ones supersede them. They differ from savepoints or snapshots by focusing on operational fault tolerance rather than operator-driven migration. Exactly-once semantics often rely on coordinated checkpoints with transactional sinks. Storage backends must be durable and performant to minimize recovery time.

ALTERNATIVE DEFINITIONS

Apache Flink

Checkpoints are consistent snapshots of the state of a streaming application, taken periodically by Flink to provide fault tolerance.

Stresses consistency, periodic capture, and fault-tolerance purpose.

Source

Apache Spark

Structured Streaming recovers progress and state by saving offsets and state to reliably replay and recompute upon failures.

Explains checkpointing as offsets and state recovery to ensure guarantees.

Source

Apache Kafka

Kafka Streams maintains local state stores with changelogs for recovery, enabling restoration after failures.

Focuses on changelog-backed state for restoration, akin to checkpointing.

Source

RELATED TERMS

Savepoint Exactly-Once Processing Durability (ACID) State Backends Recovery Offsets Idempotency

COMPETING TERMS

Snapshotting Write-Ahead Logging

RELATIONS

• Narrower: Gradient Checkpointing

KNOWLEDGE AREAS

Real-time & Streaming Data Data Engineering Data Storage & Operations MLOps

TARGET AUDIENCE

Data Engineer MLOps Engineer Data Architect DevOps Engineer

EXTERNAL SOURCES

• Wikipedia - Checkpoint (computing)
• Apache Flink - Checkpointing concepts and configuration
• Apache Spark - Recovering from failures with checkpoints
• Apache Kafka - State store recovery and changelogs

TOP QUESTIONS

How frequently should Checkpointing run in streaming jobs?

Choose an interval that balances overhead and recovery objectives. Many workloads use 30–120 seconds, tuned via testing and SLAs.

Category: Best Practices

What storage should I use for Checkpointing?

Use durable, high-throughput storage like HDFS or object stores (S3, GCS, ADLS). Avoid unstable or ephemeral disks.

Category: Requirements

How does Checkpointing enable exactly-once semantics?

By atomically capturing state and output positions, then committing sinks transactionally upon successful checkpoints to avoid duplicates.

Category: Guarantees

What are common mistakes in Checkpointing?

Storing checkpoints on unreliable disks, setting intervals too short, or not monitoring checkpoint duration and failures.

Category: Mistakes

How is Checkpointing different from Snapshotting?

Checkpoints are operational, frequent, and system-managed; snapshots may be user-initiated for migration, backups, or rollbacks.

Category: Concepts

Can Checkpointing be incremental?

Yes, many engines use incremental or copy-on-write techniques to reduce overhead and speed recovery.

Category: Implementation

What metrics should I monitor for Checkpointing?

Duration, alignment time, state size, failure rate, and end-to-end recovery time after restarts.

Category: Observability

How do backpressure and Checkpointing interact?

Backpressure can delay checkpoints; prolonged delays increase recovery point objective. Monitor and tune operator parallelism.

Category: Performance

Is Checkpointing required for at-least-once semantics?

Yes, it anchors progress to ensure replay picks up from a known consistent state after failures.

Category: Guarantees