🌊Spark Streaming

NEW

The original Spark module for processing streaming data using discretized streams (DStreams) and micro-batches.

DESCRIPTION

Spark Streaming introduced streaming to Spark via micro-batched DStreams, enabling fault-tolerant processing with checkpointing. It integrates with sources like Kafka and files. While robust, it offers less expressive APIs and is superseded by Structured Streaming, which unifies batch and streaming on DataFrames with event-time semantics. New development generally prefers Structured Streaming for simplicity and features.

ALTERNATIVE DEFINITIONS

Apache Spark

Spark Streaming makes it easy to build scalable fault-tolerant streaming applications. It provides a high-level abstraction called discretized streams (DStreams).

Introduces DStreams and emphasizes fault tolerance.

Source

Databricks

Spark Streaming processes real-time data in micro-batches using DStreams, providing integration with data sources like Kafka and HDFS.

Highlights micro-batching and common integrations.

Source

RELATED TERMS

Apache Spark Micro-batching Apache Kafka Checkpointing Event Time Watermarking Stateful Processing Backpressure

COMPETING TERMS

Apache Flink Kafka Streams Structured Streaming

RELATIONS

• Broader: Streaming

KNOWLEDGE AREAS

Real-time & Streaming Data Data Engineering Platform & Tooling

TARGET AUDIENCE

Data Engineer MLOps Engineer DevOps Engineer AI Engineer

EXTERNAL SOURCES

• Wikipedia - Apache Spark#Spark Streaming
• Apache Spark - Official Spark Streaming guide
• Apache Spark - Structured Streaming successor guide
• Apache Kafka - Kafka as a streaming source

TOP QUESTIONS

What is Spark Streaming’s core model?

It uses DStreams, representing continuous data as a sequence of micro-batches processed by Spark.

Category: Overview

How does Spark Streaming ensure fault tolerance?

It relies on lineage and checkpointing to recover state and recompute lost data.

Category: Mechanism

What sources are commonly used with Spark Streaming?

Kafka, socket streams, files, and custom receivers are typical sources.

Category: Integration

What are limitations of Spark Streaming?

Coarser micro-batch semantics and more complex APIs compared to Structured Streaming limit expressiveness and latency.

Category: Limitations

Should new projects use Spark Streaming?

New projects generally prefer Structured Streaming, which offers unified batch/stream APIs and better event-time features.

Category: Recommendation

How does micro-batching differ from true streaming?

Micro-batching processes data in small intervals, while true streaming processes event-by-event with lower latency.

Category: Concepts

How is backpressure handled in Spark Streaming?

Dynamic rate limiting and receiver-based tuning help manage ingestion under variable load.

Category: Operations

Can Spark Streaming support exactly-once semantics?

With careful sink design (e.g., idempotent writes) and checkpointing, effective exactly-once outcomes are achievable for many pipelines.

Category: Capabilities

How do you migrate from Spark Streaming to Structured Streaming?

Refactor to DataFrame/Dataset APIs, adopt event-time watermarks, and update sinks; test thoroughly with replayable sources.

Category: Migration