DataTerminology.comNEW
Events that arrive after the intended event-time window due to delays, reordering, or clock skew.
DESCRIPTION
Late data occurs when events arrive after a system’s watermark has advanced beyond their event time. Causes include network delays, retries, mobile offline periods, and GC pauses. Handling strategies combine watermarks, allowed lateness, and triggers to balance correctness and latency. Excessive lateness increases state and cost, requiring bounded retention and discard policies. Many frameworks provide metrics to observe lateness and its impact on results.
ALTERNATIVE DEFINITIONS
Apache Beam
Late data refers to data that arrives after the watermark has passed the event time of the data.
Defines lateness precisely using watermarks and event time.
Apache Spark
Data that is older than the watermark is considered ‘late’ and will be dropped from state.
Relates lateness to state eviction behavior in Structured Streaming.
RELATED TERMS
Watermarking Event Time Windowing Out-of-Order Events Allowed Lateness State Retention
COMPETING TERMS
On-time Events
RELATIONS
- Broader: Data
KNOWLEDGE AREAS
Real-time & Streaming Data Data Engineering Testing & Validation
TARGET AUDIENCE
Data Engineer MLOps Engineer Software Developer
EXTERNAL SOURCES
- Wikipedia - Stream processing
- Apache Beam - Watermarks and lateness concepts
- Apache Flink - Event time and watermarks
- Apache Spark - Handling late data with watermarking
TOP QUESTIONS
Why does Late Data happen in streaming systems?
Network variability, retries, device offline periods, and clock skew delay arrival. Distributed processing and reordering also contribute to lateness.
Category: Causes
How do watermarks relate to Late Data?
Watermarks estimate event-time completeness. Data arriving after the watermark is considered late and may be dropped or handled specially.
Category: Concept
How can I handle Late Data without losing accuracy?
Allow limited lateness, keep state longer, and use triggers to update results. Balance accuracy against latency and cost.
Category: How-to
What metrics indicate harmful Late Data levels?
Growing state size, high watermark lag, and frequent late updates signal problematic lateness. Monitor processing-time to event-time skew.
Category: Operations
Does Late Data always need to be dropped?
No. You can allow limited lateness and emit updates to previously closed windows. However, unbounded lateness is impractical.
Category: Misunderstanding
How does Late Data affect exactly-once processing?
It increases state retention and checkpoint size, but exactly-once can still be preserved with proper state management and idempotent sinks.
Category: Guarantees
What are common mistakes dealing with Late Data?
Choosing aggressive watermarks that drop useful events or unbounded lateness that explodes state. Not measuring lateness distribution is another pitfall.
Category: Mistakes
Which frameworks provide built-in Late Data handling?
Apache Beam, Flink, and Spark Structured Streaming offer watermarks, allowed lateness, and triggers or timeouts.
Category: Tools
How do mobile and IoT workloads impact Late Data?
Intermittent connectivity and batching on devices induce significant lateness. Architect for longer windows and adaptive watermarks.
Category: Use case