DataTerminology.comNEW
Also known as: Schema Validation, Data Testing
Process of checking data against rules and schemas to ensure quality and correctness.
DESCRIPTION
Data validation enforces expectations on datasets using constraints, schemas, and tests. It prevents bad data from entering pipelines and models, and provides clear failure signals. Validation spans field-level checks, relational constraints, and cross-dataset invariants. Tools like JSON Schema, Great Expectations, and dbt tests operationalize validations in CI/CD. Strong validation reduces incidents and builds trust.
ALTERNATIVE DEFINITIONS
JSON Schema
JSON Schema is a vocabulary that allows you to annotate and validate JSON documents.
Defines validation using a standard schema vocabulary for JSON.
Great Expectations
Validation is the process of checking whether data meets defined expectations, such as ranges, types, and uniqueness.
Emphasizes expectation suites for automated checking.
RELATED TERMS
Great Expectations JSON Schema Pydantic Assertion Constraints dbt Tests Unit Tests Schema Registry Extract-Transform-Load CI-CD
COMPETING TERMS
Data Verification
RELATIONS
- Broader: Validation
KNOWLEDGE AREAS
Testing & Validation Data Quality Data Engineering Data Modeling MLOps
TARGET AUDIENCE
Data Engineer Analytics Engineer Data Scientist Data Steward MLOps Engineer Software Developer
EXTERNAL SOURCES
- Wikipedia - Data validation
- JSON Schema - Guide to JSON validation
- Great Expectations - Data validation framework docs
- dbt Labs - dbt testing capabilities
TOP QUESTIONS
What is Data Validation and why do we need it?
Data Validation checks whether data meets defined rules, preventing bad data from impacting analytics and ML.
Category: Overview
What kinds of rules are used in Data Validation?
Type checks, ranges, uniqueness, referential integrity, and cross-table invariants are common.
Category: Requirements
How does Data Validation differ from Data Verification?
Validation checks conformance to rules; verification checks correctness versus a trusted source or ground truth.
Category: Distinction
Where should Data Validation run?
At ingestion, transformation, and before consumption. Integrate into CI/CD and orchestration.
Category: Best practices
What tools support Data Validation?
Great Expectations, dbt tests, JSON Schema validators, and Pydantic in Python.
Category: Overview
What are the risks of weak Data Validation?
Silent data corruption, unreliable reports, and ML model drift can result.
Category: Risk
How to design effective Data Validation suites?
Start with critical invariants, add domain rules, and maintain tests alongside models in version control.
Category: Management
How do we handle validation failures?
Fail fast with clear errors, quarantine bad data, notify owners, and provide remediation guidance.
Category: Requirements
What is the biggest misunderstanding about Data Validation?
Many believe simple null checks suffice; robust validation requires layered constraints and domain knowledge.
Category: Misunderstanding