DataTerminology.comNEW
Also known as: Data Scrubbing, Data Cleansing
Process of detecting, correcting, or removing errors, duplicates, and inconsistencies to improve dataset quality.
DESCRIPTION
Data cleaning is the practice of improving data fitness for purpose by resolving errors, duplicates, missing values, and inconsistent formats. It typically follows data profiling and precedes modeling, analytics, or reporting. Techniques include standardization, deduplication, outlier handling, validation against rules, and imputation. Effective cleaning reduces downstream rework and model bias while increasing trust and usability. It is often automated with rule engines and augmented with human review for edge cases. In modern pipelines, cleaning is codified in tests and transformations to remain reproducible and observable.
ALTERNATIVE DEFINITIONS
Wikipedia
Data cleansing, data cleaning or data scrubbing is the process of detecting and correcting (or removing) corrupt or inaccurate records from a record set, table, or database and refers to identifying incomplete, incorrect, inaccurate or irrelevant parts of the data and then replacing, modifying, or deleting the dirty or coarse data.
Highlights detection and correction/removal of corrupt or inaccurate records and handling of incomplete or irrelevant data.
IBM
Data cleansing, or data cleaning, is the process of fixing or removing incorrect, corrupted, incorrectly formatted, duplicate, or incomplete data within a dataset.
Emphasizes common error types addressed: formatting, duplication, corruption, and incompleteness.
RELATED TERMS
Data Quality Data Validation Data Preprocessing Data Enrichment Extract-Transform-Load Data Imputation Data Anomaly Detection Governance Great Expectations
COMPETING TERMS
RELATIONS
- Narrower: Address Standardization
KNOWLEDGE AREAS
Data Quality Data Management Data Engineering Data Science Testing & Validation
TARGET AUDIENCE
Data Engineer Data Scientist Analytics Engineer Data Steward BI Specialist Data Analyst
EXTERNAL SOURCES
- Wikipedia - Data cleansing
- IBM - Vendor overview of data cleansing
- Talend - Practical guide to data cleaning
- Great Expectations - Data validation framework documentation
TOP QUESTIONS
Why is Data Cleaning essential before modeling?
Data Cleaning removes errors and inconsistencies that bias models. Clean data improves accuracy, stability, and interpretability. It reduces downstream rework and increases stakeholder trust in insights.
Category: Relevancy
What are common Data Cleaning techniques?
Typical techniques include deduplication, standardization, type casting, outlier handling, missing value imputation, and constraint-based validation. Many teams codify these as repeatable transformations.
Category: Example
How does Data Cleaning differ from Data Profiling?
Data Profiling measures and describes data characteristics, while Data Cleaning fixes issues discovered. Profiling informs cleaning rules; cleaning applies corrective actions.
Category: Distinction
What risks exist when skipping Data Cleaning?
Unclean data leads to faulty analyses, biased models, and operational incidents. It erodes trust and can cause compliance failures if invalid data propagates.
Category: Risk
Which tools support automated Data Cleaning?
Common tools include dbt tests, Great Expectations, Deequ, data quality platforms, and validation in ETL/ELT frameworks like Airflow or Spark.
Category: Tools
How do Data Cleaning rules relate to data contracts?
Data contracts often encode schema and quality expectations. Cleaning enforces or remediates against those expectations to keep data consumer-ready.
Category: Concept
When should Data Cleaning be applied in a pipeline?
Apply cleaning after ingestion and profiling, before analytics or modeling. For streaming, implement incremental validations and corrections in-flight.
Category: Practice
What is a common mistake in Data Cleaning?
Overly aggressive filtering can discard valuable signal. Always quantify impact and document rationale for transformations.
Category: Mistakes
How to measure effectiveness of Data Cleaning?
Track quality metrics like completeness, validity, uniqueness, and error rates before and after cleaning. Establish SLAs and monitor trends over time.
Category: Measurement