DataTerminology.comNEW
Identifying and removing duplicate records to improve data quality, storage efficiency, and reliable analytics.
DESCRIPTION
Deduplication detects exact or near-duplicate records and merges or removes them to ensure single representations. Techniques range from hash-based comparisons to fuzzy matching and probabilistic linkage. It underpins identity resolution and clean master data. Deduplication reduces storage costs, improves performance, and prevents double-counting. Governance defines survivorship rules and audit trails for merges.
ALTERNATIVE DEFINITIONS
Wikipedia
Data deduplication is a technique for eliminating duplicate copies of repeating data in storage.
Storage-focused definition highlighting elimination of duplicates.
Informatica
Data deduplication identifies and removes duplicate records by matching and consolidating entries to create a single, accurate view.
Emphasizes record-level matching and consolidation for master data.
RELATED TERMS
Uniqueness Merge-Purge Canonicalization Weighted Approximate Matching Hashing Bloom Filter Primary Key Constraint Master Data Management
COMPETING TERMS
RELATIONS
- Narrower: Data Deduplication
KNOWLEDGE AREAS
Data Quality Data Engineering Data Management
TARGET AUDIENCE
Data Engineer Data Steward Data Analyst Data Modeler
EXTERNAL SOURCES
- Wikipedia - Data deduplication
- Apache Spark - Drop duplicates API reference
- Google Cloud - Record linkage solution guide
- AWS - Glue data quality rules
TOP QUESTIONS
Why is Deduplication necessary?
Duplicates distort metrics, inflate costs, and complicate identity management. Deduplication ensures clean counts and accurate analytics.
Category: Relevancy
How do you perform Deduplication?
Use deterministic keys, hashing, and fuzzy matching. Apply merge-purge workflows with survivorship rules and audit trails.
Category: Requirements
What mistakes happen in Deduplication?
Over-aggressive merges that collapse distinct entities, and under-detection of near duplicates due to weak matching rules.
Category: Mistakes
What’s the risk of poor Deduplication?
Lost distinct records, corrupted master data, and reporting errors. It can damage customer experience and compliance.
Category: Risks
What are alternatives or related techniques to Deduplication?
Entity resolution and record linkage unify identities across sources; canonicalization standardizes formats before matching.
Category: Alternatives
How to manage Deduplication over time?
Implement periodic scans, monitor duplicate rates, and refine matching rules as data evolves. Maintain lineage of merges.
Category: Management
Where to start with Deduplication?
Profile datasets, identify key fields, and run initial duplicate detection to quantify scope. Prioritize high-impact domains.
Category: Topic
What’s a misunderstanding about Deduplication?
It’s not only exact duplicates; near duplicates and cross-system identity challenges demand more sophisticated matching.
Category: Misunderstanding
What’s a hot topic around Deduplication?
ML-based entity resolution using embeddings and graph algorithms scales dedup across large, noisy datasets.
Category: Topic