🌊Structured Streaming

NEW

A declarative, scalable stream processing engine built on Spark SQL.

DESCRIPTION

Structured Streaming lets you write streaming queries like batch SQL/DataFrame transformations and runs them incrementally. It supports micro-batch and continuous processing, event-time semantics, watermarks, and stateful aggregations. Sources and sinks include Kafka, files, and Delta Lake with checkpointing for fault tolerance. Exactly-once semantics are achievable for supported sinks using idempotence and transactional writes. It’s widely used for real-time ETL, monitoring, and streaming analytics on the Spark platform.

ALTERNATIVE DEFINITIONS

Apache Spark

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine.

Stresses scalability, fault tolerance, and Spark SQL foundation.

Source

Databricks

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine.

Vendor documentation echoing the core definition with platform specifics elsewhere.

Source

RELATED TERMS

Apache Spark Apache Kafka Watermarking Event Time Micro Batch Processing Stateful Processing Delta Lake Exactly-Once Processing

COMPETING TERMS

Apache Flink Kafka Streams

RELATIONS

• Broader: Streaming
• Narrower: Spark Structured Streaming

KNOWLEDGE AREAS

Real-time & Streaming Data Data Engineering Platform & Tooling Data Science

TARGET AUDIENCE

Data Engineer MLOps Engineer AI Engineer Data Architect Software Developer

EXTERNAL SOURCES

• Wikipedia - Apache Spark
• Apache Spark - Official programming guide
• Databricks - Kafka source and sink guidance
• Delta Lake - Delta sinks with transactional guarantees

TOP QUESTIONS

How does Structured Streaming differ from Spark Streaming (DStreams)?

Structured Streaming is declarative and built on Spark SQL, enabling optimizations and unified batch/stream APIs. DStreams are lower-level micro-batch primitives now largely superseded.

Category: Comparison

What is watermarking in Structured Streaming?

Watermarking bounds state by specifying how late data can arrive for event-time aggregations. It permits discarding old state while tolerating limited lateness.

Category: Concept

Can Structured Streaming achieve exactly-once semantics?

Yes, for supported sinks via idempotent writes and transactional commits combined with checkpointing. Kafka and Delta sinks are common exactly-once scenarios.

Category: Guarantees

When should I choose continuous processing over micro-batch?

Choose continuous processing for ultra-low latency when supported; micro-batch suits most workloads balancing latency, throughput, and reliability.

Category: Guidance

How is state managed for aggregations?

State is stored and checkpointed to fault-tolerant storage. Watermarks and timeouts bound state to control memory and performance.

Category: Architecture

What sources and sinks does Structured Streaming support?

Common ones include Kafka, files (Parquet/JSON), and Delta Lake. Third-party connectors extend to Kinesis and other systems.

Category: Capabilities

How do I handle late data in Structured Streaming?

Use withWatermark on the event-time column and define allowed lateness for windowed aggregations. Tune to balance correctness and resource usage.

Category: How-to

What are common performance pitfalls?

Too-small micro-batch intervals, unbounded state from missing watermarks, and skewed keys cause bottlenecks. Optimize shuffle partitions and use trigger tuning.

Category: Mistakes

How do I monitor and debug Structured Streaming jobs?

Leverage Spark UI, metrics, and logs, plus query progress events. Store checkpoints reliably and instrument lag and watermark metrics.

Category: Operations