💧Watermark

NEW

A heuristic threshold indicating event-time progress, used to decide when to emit results despite out-of-order data.

DESCRIPTION

Watermarks represent the system’s belief that all events with timestamps earlier than the watermark have likely arrived. They allow windowed computations to produce timely results while tolerating late data. Watermarks advance based on source timestamps and heuristics, and can be idleness-aware. Late events can still update results if allowed lateness is configured. Proper watermarking balances freshness and correctness in event-time processing.

ALTERNATIVE DEFINITIONS

Apache Flink

Watermarks are the mechanism in Flink to measure progress in event time and to handle late data.

Emphasizes event-time progress and late-data handling in Flink.

Source

Google Cloud

A watermark is a threshold that indicates the system’s progress in processing data in event time.

Describes watermark as a progress threshold across event time for streaming pipelines.

Source

Apache Spark

Watermarking is a technique to deal with late data by specifying how late data is expected to be and discarding state for old windows.

Explains watermarks in the context of state eviction and late data in Spark.

Source

RELATED TERMS

Event Time Late Data Windowing Triggers Out-of-Order Events Allowed Lateness Idling Sources Stream-time Session Window Timestamps

COMPETING TERMS

Processing-Time Semantics Punctuation (Streams)

KNOWLEDGE AREAS

Real-time & Streaming Data Data Engineering Data Science

TARGET AUDIENCE

Data Engineer Data Scientist MLOps Engineer Software Developer

EXTERNAL SOURCES

• Wikipedia - Stream processing
• Google Cloud - Dataflow watermarks guide
• Apache Flink - Event time and watermarks
• Apache Spark - Late data and watermarking

TOP QUESTIONS

How do Watermarks handle out-of-order events?

They define a moving event-time boundary; events older than the watermark are considered late and may update results within an allowed lateness interval.

Category: Concepts

How are Watermarks computed?

Commonly via source timestamps, max-observed minus delay, per-partition aggregation, and idleness detection.

Category: Implementation

What’s the tradeoff in setting watermark lateness?

More lateness improves correctness for late events but increases state size and latency.

Category: Tradeoffs

Do Watermarks guarantee no more late data?

No. They are heuristics; truly late events can still arrive and must be handled as updates or drops.

Category: Misunderstanding

How do Watermarks interact with triggers?

Triggers often fire when watermarks pass window ends, emitting on-time results and optional updates for late data.

Category: Concepts

How to monitor Watermarks?

Track watermark progression, idle partitions, and late-data rates to detect skew and misconfiguration.

Category: Observability

What are common mistakes with Watermarks?

Using processing-time clocks, ignoring idleness, and applying a one-size-fits-all delay across heterogeneous sources.

Category: Mistakes

When not to use Watermarks?

If only processing-time metrics matter or events lack reliable timestamps, watermarks add complexity without benefit.

Category: Usage

What is a hot topic for Watermarks?

Adaptive watermarks per partition based on observed lateness distributions are emerging.

Category: Topic