DataTerminology.comNEW
Automated checkpoints that block, quarantine, or flag bad data based on defined rules and thresholds.
DESCRIPTION
Data quality gates are automated controls embedded in data pipelines that evaluate records against expectations before data flows downstream. They enforce constraints like schema, nullability, ranges, referential integrity, and distributional properties. Gates can pass, quarantine, or fail pipelines depending on severity and policies. They reduce downstream breakage, improve trust, and support SLAs for analytics, ML features, and APIs. Modern platforms let teams declare expectations as code and integrate with lineage, observability, and incident workflows.
ALTERNATIVE DEFINITIONS
AWS Labs Deequ
Deequ is a library built on top of Apache Spark for defining ‘unit tests for data’, which measure data quality in large datasets.
Introduces test-like checks as code for scalable quality measurement.
Great Expectations
An Expectation is a verifiable assertion about your data.
Defines checks as explicit, verifiable assertions.
Google Cloud Dataform
An assertion is a SQL query that returns zero rows when your data is valid.
Expresses validation as SQL predicates that must not return failures.
SonarQube
A Quality Gate is a set of conditions that must be met before code can be promoted to production.
Provides the gating concept from software quality, analogous to data pipelines.
RELATED TERMS
Data Quality Data Validation Schema Enforcement Unit Tests for Data Data Anomaly Detection Data Observability Great Expectations AWS Deequ Delta Live Tables
COMPETING TERMS
Data Monitoring Data Contracts
KNOWLEDGE AREAS
Data Quality Data Engineering Testing & Validation MLOps Observability & Telemetry
TARGET AUDIENCE
Data Engineer Analytics Engineer Data Steward MLOps Engineer Data Architect
EXTERNAL SOURCES
- Wikipedia - Data quality
- AWS Labs Deequ - Open-source data quality checks library
- Great Expectations - Data validation with expectations
- Databricks - Pipeline expectations and enforcement
- Google Cloud Dataform - SQL assertions for data pipelines
TOP QUESTIONS
Why are Data Quality Gates critical in modern pipelines?
They stop bad data early, preventing downstream breakages, costly reprocessing, and mistrust. Gates enforce SLAs and reduce incident frequency.
Category: Relevancy
How do Data Quality Gates differ from data observability?
Gates enforce preventive rules during processing; observability passively monitors and detects anomalies post-factum. Both are complementary.
Category: Misunderstanding
What types of checks belong in Data Quality Gates?
Schema conformance, null thresholds, domain/range checks, referential integrity, distributional drift, and deduplication rules are common.
Category: Example
Where should Data Quality Gates be placed?
At ingestion, before critical joins/aggregations, and at publish boundaries to protect downstream consumers.
Category: Best practices
How are Data Quality Gates managed as code?
Define rules in configuration or code (YAML/SQL/Python), version them in Git, validate in CI, and deploy with pipelines.
Category: Management
What happens when a Data Quality Gate fails?
Pipelines may fail fast, quarantine records, or log and continue depending on severity policies and consumer impact.
Category: Risks
What tools implement Data Quality Gates?
Great Expectations, AWS Deequ, dbt tests, Databricks DLT expectations, and BigQuery/Dataform assertions are popular.
Category: Tools
How do Data Quality Gates relate to data contracts?
Contracts define producer-consumer guarantees; gates enforce those guarantees programmatically at runtime.
Category: Relationship
What are common mistakes with Data Quality Gates?
Overly rigid rules, ignoring data evolution, and not distinguishing warn vs. fail severity can create false positives and toil.
Category: Mistakes