DataTerminology.comNEW
A Spark-based library for defining and evaluating data quality constraints and metrics at scale.
DESCRIPTION
Deequ, developed by AWS Labs, provides a declarative API to define ‘unit tests for data’ as constraints over large datasets. It computes metrics, stores them over time, and detects anomalies. Deequ integrates with Apache Spark, enabling scalable checks on data lakes and ETL pipelines. It supports constraint suggestion and incremental metrics. Deequ is well-suited for JVM stacks and big data workloads.
ALTERNATIVE DEFINITIONS
AWS Labs (GitHub)
Deequ is a library built on top of Apache Spark for defining “unit tests for data”, which measure data quality in large datasets.
Highlights the ‘unit tests for data’ approach and Spark foundation.
AWS Big Data Blog
Deequ is an open source library that allows you to define data quality constraints on your data and run these checks on Spark to ensure data quality at scale.
Emphasizes scale and Spark execution for checks.
RELATED TERMS
Data Quality Data Validation Apache Spark Constraints Metrics Repository Data Anomaly Detection AWS Glue Data Profiling Unit Tests for Data
COMPETING TERMS
KNOWLEDGE AREAS
Data Quality Data Engineering Platform & Tooling Data Storage & Operations Testing & Validation
TARGET AUDIENCE
Data Engineer MLOps Engineer Analytics Engineer Data Steward Data Architect
EXTERNAL SOURCES
- Wikipedia - Deequ
- AWS Labs - Official repository and README
- AWS - Intro and examples on Deequ
- Apache Spark - Spark docs for Deequ runtime
TOP QUESTIONS
What is Deequ used for?
Deequ defines and validates data quality constraints on large datasets using Spark, enabling scalable, automated checks in data pipelines.
Category: Definition
How does Deequ scale data quality checks?
It runs computations on Apache Spark, distributing metric calculation and constraint evaluation across clusters for big data workloads.
Category: Capabilities
What are constraints in Deequ?
Constraints are declarative assertions about data, such as completeness, uniqueness, or value ranges, evaluated against computed metrics.
Category: Concepts
Can Deequ detect anomalies over time?
Yes, Deequ stores historical metrics and supports anomaly detection to identify unexpected regressions in quality.
Category: Monitoring
How does Deequ compare to Great Expectations?
Deequ is Spark-first and JVM-based, ideal for big data. Great Expectations offers rich docs and multi-engine support; choice depends on tech stack.
Category: Comparison
Does Deequ integrate with AWS services?
It commonly runs with AWS Glue, EMR, and S3, fitting naturally into AWS-centric data lakes.
Category: Integration
What is a common mistake when using Deequ?
Computing all metrics on full datasets can be costly. Use sampling or incremental patterns where appropriate.
Category: Mistakes
How do you start with Deequ?
Add the dependency to your Spark project, define constraints with the VerificationSuite, and run checks in your ETL jobs.
Category: Getting Started
Can Deequ suggest constraints?
Yes, it can profile datasets to propose initial constraints, which you should review and refine.
Category: Features