⚡Spark SQL

NEW

An Apache Spark module for structured data processing using SQL, DataFrames, and Dataset APIs with a cost-based optimizer.

DESCRIPTION

Spark SQL unifies relational processing with Spark’s distributed engine through DataFrames and SQL queries. The Catalyst optimizer plans queries, while Tungsten enhances memory and code generation. It reads/writes formats like Parquet, ORC, and JSON, and integrates with Hive metastore and JDBC sources. It powers ETL, ELT, BI, and ML feature pipelines at scale. Structured Streaming extends the same APIs to streaming workloads.

ALTERNATIVE DEFINITIONS

Apache Spark

Spark SQL is a Spark module for structured data processing. It provides a programming abstraction called DataFrames and can also act as a distributed SQL query engine.

Introduces DataFrames and SQL query capabilities together.

Source

Databricks

Spark SQL provides a unified way to process structured and semi-structured data using SQL, DataFrames, and Datasets on Apache Spark.

Emphasizes unified processing of structured and semi-structured data.

Source

RELATED TERMS

Apache Spark DataFrame Catalyst Optimizer Tungsten SQL Parquet Delta Lake Java Database Connectivity

COMPETING TERMS

Apache Hive Trino

RELATIONS

• Broader: SQL
• Conforms to: Java Database Connectivity
• Narrower: Apache Spark SQL

KNOWLEDGE AREAS

Data Engineering Data Warehousing & BI Data Science Platform & Tooling

TARGET AUDIENCE

Data Engineer Data Scientist Analytics Engineer BI Specialist

EXTERNAL SOURCES

• Wikipedia - Apache Spark#Spark SQL
• Apache Spark - Official Spark SQL overview
• Apache Spark - SQL programming guide
• Databricks - Databricks Spark SQL docs

TOP QUESTIONS

What does Spark SQL add to Apache Spark?

It provides SQL query capabilities and DataFrame/Dataset APIs with a cost-based optimizer for efficient structured data processing.

Category: Overview

How does Catalyst optimize queries in Spark SQL?

Catalyst builds logical and physical plans, applies rule- and cost-based optimizations, and selects efficient execution strategies.

Category: Mechanism

What data formats work well with Spark SQL?

Columnar formats like Parquet and ORC are preferred for predicate pushdown and compression; JSON and CSV are also supported.

Category: Best Practices

How does Spark SQL integrate with the Hive Metastore?

It can read Hive tables, leverage catalogs, and run SQL with compatible semantics for unified access.

Category: Integration

When should I use DataFrames vs. Datasets?

DataFrames (untyped) are concise and optimized; Datasets add type safety in Scala/Java with some overhead.

Category: Decision

How does Spark SQL compare to Trino?

Spark SQL excels at large-scale ETL and batch processing integrated with Spark; Trino focuses on federated, interactive SQL across many sources.

Category: Comparison

Can Spark SQL handle semi-structured data?

Yes. Nested schemas, JSON functions, and UDFs enable processing of semi-structured data.

Category: Capabilities

What are common Spark SQL performance pitfalls?

Skewed joins, small files, missing partitioning, and excessive UDFs often slow queries. Optimize partitions, statistics, and broadcast joins.

Category: Mistakes

How does Structured Streaming relate to Spark SQL?

Structured Streaming uses the same DataFrame/Dataset APIs and engine, extending Spark SQL to streaming data.

Category: Integration