DataTerminology.comNEW
Also known as: Metadata Ingestion, Metadata Crawling, Catalog Harvesting
Metadata harvesting automatically collects metadata from systems, repositories, and data sources to populate catalogs and keep them synchronized.
DESCRIPTION
Metadata harvesting enables scalable discovery and governance by programmatically collecting technical, operational, and business metadata across platforms. Common approaches include API-based extraction, connectors, crawling, and standards-driven protocols like OAI-PMH and DCAT. Harvesters normalize and map fields to catalog schemas, deduplicate records, and handle incremental updates. They support federation by aggregating multiple catalogs into a unified view. Effective harvesting reduces manual documentation effort and improves accuracy and freshness of metadata at enterprise scale.
ALTERNATIVE DEFINITIONS
W3C
Data catalogs can be federated by aggregating catalogs, harvesting metadata from multiple catalogs, or a combination of these approaches.
Highlights harvesting as a federation approach in DCAT.
Open Archives Initiative
The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) provides an application-independent interoperability framework based on metadata harvesting.
Defines a protocol specifically for metadata harvesting interoperability.
CKAN
The CKAN Harvester extension allows metadata to be pulled from other catalogs and imported into CKAN.
Shows practical catalog-to-catalog harvesting implementation.
RELATED TERMS
Data Discovery Platform DCAT OAI-PMH Data Exploration API Integration Lineage Schema Registry Extract-Transform-Load Data Integration Web Crawling
COMPETING TERMS
Manual Metadata Curation
KNOWLEDGE AREAS
Metadata Management Data Management Platform & Tooling Data Integration & Interoperability
TARGET AUDIENCE
Data Steward Data Engineer Information Analyst Data Architect
EXTERNAL SOURCES
- Wikipedia - Open Archives Initiative Protocol for Metadata Harvesting
- W3C - DCAT standard on catalog federation
- CKAN - Open-source harvester implementation
- Open Archives Initiative - OAI-PMH protocol specification
TOP QUESTIONS
Why use Metadata Harvesting instead of manual entry?
Automation scales across thousands of assets, reduces errors, and keeps catalogs fresh. Manual entry is brittle and quickly becomes outdated.
Category: Relevancy
Which standards support Metadata Harvesting?
OAI-PMH enables repository harvesting; DCAT supports catalog federation and exchange. Many platforms offer REST APIs for extraction.
Category: Overview
How do you handle duplicate metadata during harvesting?
Use unique identifiers, matching rules, and reconciliation logic to merge duplicates and preserve provenance.
Category: Requirements
What are common mistakes in Metadata Harvesting?
Failing to normalize fields, poor incremental update strategies, and ignoring access controls can lead to messy catalogs and compliance issues.
Category: Mistakes
What’s the risk of aggressive Metadata Harvesting?
Excessive crawling may overload systems or expose sensitive details. Respect rate limits and apply redaction policies.
Category: Risk
How does Metadata Harvesting relate to lineage?
Harvested metadata can include job runs and dependencies, which feed lineage graphs with accurate upstream and downstream links.
Category: Example
Who should own Metadata Harvesting?
Data stewards define governance and mapping rules, while engineers implement connectors and schedules; architects set standards.
Category: Management
How often should Metadata Harvesting run?
Frequency depends on change velocity—ranging from near real-time events to nightly batches—balanced with rate limits and freshness needs.
Category: Opinion
What tools support Metadata Harvesting?
Catalogs like CKAN, OpenMetadata, and DataHub provide harvesters; vendors offer connectors and ingestion APIs.
Category: Overview