DataTerminology.comNEW
Also known as: Data Quality Monitoring, Pipeline Monitoring
Continuous tracking of dataset and pipeline health using metrics, checks, and alerts to detect issues early.
DESCRIPTION
Data monitoring ensures data assets meet expectations for freshness, completeness, accuracy, and volume. It uses metrics, rules, and anomaly detection to surface regressions and drift. Integrations with orchestration and catalogs enable context and ownership-aware alerting. Monitoring reduces incident impact by shrinking detection and response times. Mature programs track SLO compliance and align alerts to business impact. As practices evolve, teams extend monitoring with lineage, tracing, and automated remediation.
ALTERNATIVE DEFINITIONS
Wikipedia
Application performance management (APM) is the monitoring and management of performance and availability of software applications.
Describes a monitoring subset focused on performance and availability; analogous patterns apply to data systems.
Wikipedia
Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.
Frames monitoring within the broader goal of understanding system state via signals.
RELATED TERMS
Measure Logging Tracing Data Quality Data Anomaly Detection Alerting Lineage SLA Great Expectations
COMPETING TERMS
Data Observability
RELATIONS
- Broader: Monitoring
KNOWLEDGE AREAS
Observability & Telemetry Data Engineering MLOps Data Quality Testing & Validation
TARGET AUDIENCE
Data Engineer MLOps Engineer Analytics Engineer Data Steward Data Architect BI Specialist
EXTERNAL SOURCES
- Wikipedia - Observability
- Google Cloud - Data observability reference architecture
- Microsoft - Monitor data pipelines in ADF
- Great Expectations - Data validation and monitoring concepts
- Datadog - Guide to monitoring data pipelines
TOP QUESTIONS
What metrics matter most in Data Monitoring?
Track freshness, volume, completeness, accuracy proxies, schema change rate, and error counts. Include latency and throughput for pipelines. Align thresholds to business impact and SLOs.
Category: Requirements
How does Data Monitoring differ from Data Observability?
Monitoring emphasizes metrics and alerts, while observability adds rich context like lineage, logs, and traces to explain issues. Observability enables faster root cause analysis.
Category: Concept
What tools are commonly used for Data Monitoring?
Observability platforms, validation frameworks, orchestration UIs, and metric stores (e.g., Prometheus, Cloud Monitoring). Integration with incident management is essential.
Category: Tools
How do I start Data Monitoring effectively?
Begin with key datasets and critical SLOs. Add checks at data ingress and model inputs. Iterate by triaging alert noise and automating runbooks.
Category: HowTo
What are common mistakes in Data Monitoring?
Alert floods without prioritization, siloed dashboards, missing ownership, and ignoring business context. Lack of schema-change detection causes blind spots.
Category: Mistakes
Why is Data Monitoring important for MLOps?
It catches data issues before they degrade model performance. Monitoring features and drift is crucial to maintain prediction quality.
Category: Relevancy
How should alerts be routed in Data Monitoring?
Route by ownership from catalog/lineage, include run context and impact, and integrate on-call escalation with quiet hours and deduplication.
Category: BestPractices
What is a hot topic in Data Monitoring today?
Lineage-aware and LLM-assisted root cause analysis are gaining adoption. Data contracts are also tightening monitoring scope.
Category: Topic
How does testing relate to Data Monitoring?
Tests prevent regressions at development time; monitoring detects runtime drift and anomalies. Both are complementary for reliability.
Category: Testing