DataTerminology.comNEW
Also known as: Data Scrubbing
The process of detecting and correcting or removing errors, inconsistencies, and duplicates in data.
DESCRIPTION
Scrubbing improves data fitness for purpose by fixing or removing inaccurate, inconsistent, or incomplete records. Techniques include profiling, validation rules, standardization, deduplication, and missing value imputation. It is often executed within ETL/ELT pipelines and governed by data quality frameworks. Scrubbing reduces downstream rework and analytics risk, but must be transparent and auditable. Ongoing monitoring complements periodic cleansing to sustain quality.
ALTERNATIVE DEFINITIONS
Wikipedia
Data cleansing, also known as data cleaning or data scrubbing, is the process of detecting and correcting (or removing) corrupt or inaccurate records from a record set, table, or database.
Explicitly equates data scrubbing with cleansing and cleaning, emphasizing correction/removal.
IBM
Data cleansing is the process of fixing or removing incorrect, corrupted, incorrectly formatted, duplicate, or incomplete data within a dataset.
Enumerates typical quality issues addressed during cleansing.
RELATED TERMS
Data Cleaning Data Quality Data Profiling Data Validation Feature Scaling Entity Matching Imputation Data Enrichment Extract-Transform-Load Curation
COMPETING TERMS
KNOWLEDGE AREAS
Data Quality Data Management Data Engineering Metadata Management
TARGET AUDIENCE
Data Steward Data Engineer Data Analyst Data Management Consultant Data Governance Manager
EXTERNAL SOURCES
- Wikipedia - Data cleansing
- Gretel - Data transform and cleansing reference
- Microsoft Learn - Data flow transformations for cleaning
TOP QUESTIONS
Why is Scrubbing essential before analytics?
It prevents misleading insights by ensuring accurate, consistent data. Clean data reduces rework and fosters trust.
Category: Relevancy
What’s the difference between Scrubbing and profiling?
Profiling discovers data issues; scrubbing fixes them. They complement each other in a quality lifecycle.
Category: Definition
What are common mistakes in Scrubbing?
Silently altering data without lineage, over-aggressive deduplication, and inconsistent rule application can erode trust.
Category: Mistakes
How do I prioritize Scrubbing rules?
Start with high-impact dimensions (accuracy, completeness), align with business requirements, and iterate based on defect rates.
Category: Management
What tools assist with Scrubbing at scale?
Use data quality frameworks, ETL tools, and libraries like Great Expectations for validations integrated into pipelines.
Category: Implementation
What are risks of excessive Scrubbing?
Overfitting to known issues can hide systemic problems. Data loss from aggressive filtering can bias analyses.
Category: Risks
How does Scrubbing relate to master data management?
Scrubbing improves instance quality; MDM ensures authoritative records and survivorship rules across sources.
Category: Definition
What is a hot topic around Scrubbing?
Automated anomaly detection and ML-assisted matching are enhancing traditional cleansing workflows.
Category: Topic
Where should teams start with Scrubbing?
Profile critical datasets, define measurable quality rules, and build automated checks into CI/CD for data.
Category: Learning