DataTerminology.comNEW
Documentation of data’s origins, movements, and transformations from source to consumption for transparency and impact analysis.
DESCRIPTION
Data lineage captures how data flows through systems, including sources, transformations, joins, and destinations. It supports trust, debugging, impact analysis, and compliance by making dependencies visible. Lineage can be captured at job, column, or field level using instrumentation and metadata harvesting. Visual graphs help understand blast radius of changes and trace errors. Standardized approaches like OpenLineage enable interoperability. Accurate lineage underpins responsible analytics and governance.
ALTERNATIVE DEFINITIONS
Wikipedia
Data lineage includes the data’s origins, what happens to it, and where it moves over time.
Concise summary covering origin, transformation, and movement over time.
IBM
Data lineage is the process of understanding, recording, and visualizing data as it flows from data sources to consumption.
Emphasizes recording and visualization across the flow from source to consumption.
RELATED TERMS
Metadata Management Data Discovery Platform Impact Analysis Data Observability Data Quality Extract-Transform-Load ELT Governance OpenLineage Lineage Graph
COMPETING TERMS
Black-box Processing
RELATIONS
- Broader: Lineage
KNOWLEDGE AREAS
Metadata Management Data Governance Observability & Telemetry Data Engineering Data Management
TARGET AUDIENCE
Data Steward Data Engineer Data Architect BI Specialist Chief Data Officer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Data lineage
- OpenLineage - Standard for lineage metadata
- Google Cloud - Data lineage in Google Cloud
- Microsoft Purview - Lineage collection guidance
TOP QUESTIONS
Why is Data Lineage important?
It enables trust, debugging, and change impact analysis by revealing how data is produced and transformed across systems.
Category: Relevancy
What levels of granularity exist in Data Lineage?
Job-level, dataset-level, and column/field-level lineage provide increasingly detailed visibility and traceability.
Category: Concept
How is Data Lineage captured?
Through pipeline instrumentation, query parsing, metadata harvesting, and standardized events (e.g., OpenLineage).
Category: Practice
What is a common mistake with Data Lineage?
Collecting lineage without ensuring accuracy or usability—noisy, incomplete graphs hinder decisions.
Category: Mistakes
How does Data Lineage support compliance?
It proves data sources, transformations, and access paths for audits and regulatory reporting.
Category: Compliance
What tools provide Data Lineage?
Catalogs, observability platforms, and orchestration tools with lineage integration; OpenLineage fosters interoperability.
Category: Tools
How do you validate Data Lineage quality?
Compare lineage with code repositories, run spot checks on transformations, and monitor collection coverage.
Category: Quality
How does Data Lineage assist in incident response?
It quickly identifies affected downstream assets and upstream root causes to accelerate remediation.
Category: Example
Can Data Lineage be automated?
Yes. Instrumentation and metadata pipelines automate capture; human curation refines ambiguous edges.
Category: BestPractice