DataTerminology.comNEW
Duplicate identification detects multiple records representing the same entity to improve quality and enable consolidation.
DESCRIPTION
Duplicate identification locates records that refer to the same entity within or across datasets. It is a core step in deduplication, matching, and master data management. Approaches include deterministic keys, probabilistic scoring, and ML transforms. Detection triggers merge workflows and survivorship policies to form a golden record. Reducing duplicates improves analytics accuracy, customer experience, and operational efficiency.
ALTERNATIVE DEFINITIONS
Wikipedia
Record linkage is the task of finding records in a data set that refer to the same entity across different data sources.
Explains duplicate detection as part of linking records to the same entity.
AWS Glue
FindMatches is a machine learning transform that finds duplicate records in your dataset.
Describes managed ML approach to detecting duplicates.
Salesforce
Duplicate Management helps you prevent users from creating duplicates in Salesforce.
Focuses on application-level prevention and detection of duplicates.
RELATED TERMS
Matching Entity Matching Entity Resolution Single Source of Truth Survivorship Rules Data Quality Blocking Weighted Approximate Matching Merge Policies
COMPETING TERMS
RELATIONS
- Broader: Identification
KNOWLEDGE AREAS
Data Management Data Quality Reference & Master Data Data Integration & Interoperability
TARGET AUDIENCE
Data Steward Data Engineer Data Architect Data Modeler Data Analyst
EXTERNAL SOURCES
- Wikipedia - Record linkage
- Salesforce - Duplicate management features overview
- OpenRefine - Clustering to find duplicate values
- AWS Glue - ML transform for duplicate detection
- IBM - Suspect duplicate identification process
TOP QUESTIONS
What is Duplicate Identification?
It detects records representing the same entity to improve quality and support merge workflows.
Category: Definition
Why is Duplicate Identification important?
Duplicates inflate metrics and cause inconsistent experiences. Identifying them improves analytics and operations.
Category: Relevancy
How do you identify duplicates?
Apply deterministic keys, probabilistic scoring, and similarity measures, often with blocking to reduce comparisons.
Category: Implementation
What risks exist in Duplicate Identification?
False merges corrupt data; missed duplicates undermine decisions. Stewardship and audit mitigate these risks.
Category: Risk
What metrics evaluate Duplicate Identification?
Precision, recall, F1, and pair quality gauge success; monitor over time for drift.
Category: Metrics
How does Duplicate Identification relate to Identity Resolution?
It’s a subset focusing on within-entity duplicates; identity resolution spans cross-system unification.
Category: Distinction
What data prep helps Duplicate Identification?
Standardization, parsing addresses, phonetic encoding, and deduped identifiers improve detection.
Category: Preparation
Should Duplicate Identification be automated?
Automate high-confidence cases; route borderline pairs to stewards for review.
Category: BestPractices
How do merge policies follow Duplicate Identification?
Survivorship rules select attribute values from sources, forming a trusted golden record.
Category: Management