DataTerminology.comNEW
Also known as: ML Monitoring
The practice of tracking models’ performance, data quality, and drift in production to detect issues early.
DESCRIPTION
Model monitoring continuously evaluates production models to detect drift, performance degradation, and data quality issues. It compares live inputs and predictions against training baselines and business KPIs. Monitoring systems compute statistical tests, fairness metrics, and error analysis, triggering alerts when thresholds are exceeded. They capture logs, traces, and structured telemetry to support root-cause analysis. Monitoring integrates with retraining workflows and champion–challenger deployments. Clear dashboards and SLOs help stakeholders manage model health. Proper monitoring reduces risk, ensures compliance, and maintains user trust.
ALTERNATIVE DEFINITIONS
Google Cloud
Model monitoring lets you automatically monitor input data and predictions for drift and anomalies.
Automated monitoring of inputs and predictions with drift/anomaly focus.
AWS
Amazon SageMaker Model Monitor automatically monitors machine learning models in production.
Emphasizes automation for production models.
Evidently AI
Model monitoring is the practice of tracking the performance of machine learning models in production.
Highlights ongoing tracking of performance in production settings.
RELATED TERMS
Concept Drift Data Drift Model Drift Observability Model Registry A/B Testing Alerting Explainability Data Quality
COMPETING TERMS
Model Validation
RELATIONS
- Broader: Monitoring
KNOWLEDGE AREAS
MLOps Observability & Telemetry Testing & Validation AI Data Science
TARGET AUDIENCE
MLOps Engineer AI Engineer Data Scientist Data Engineer Data Governance Manager Chief Data Officer AI Product Manager
EXTERNAL SOURCES
- Wikipedia - Concept drift
- Google Cloud - Vertex AI Model Monitoring
- AWS - SageMaker Model Monitor docs
- Microsoft - Azure ML model monitoring concepts
TOP QUESTIONS
What metrics are essential for Model Monitoring?
Track prediction accuracy, business KPIs, data drift, concept drift, latency, fairness metrics, and model confidence calibration. Coverage and missingness are also critical.
Category: Requirements
How does Model Monitoring detect drift?
It compares live distributions to baselines using statistical tests (e.g., KS, PSI), monitoring feature and prediction shifts over time.
Category: Application
What is a common misunderstanding about Model Monitoring?
Many assume offline validation is sufficient. In reality, data and behavior evolve, so continuous monitoring is necessary to catch real-world changes.
Category: Misunderstanding
How is Model Monitoring integrated with retraining?
Alerts can trigger retraining pipelines, schedule data labeling, or launch champion–challenger evaluations before promoting a new model.
Category: Management
What are the risks of not monitoring models?
Silent model decay can harm users and business outcomes, cause fairness violations, and lead to regulatory non-compliance.
Category: Risks
How does Model Monitoring support compliance?
It provides audit logs, documented thresholds, fairness reports, and incident records that demonstrate control and oversight.
Category: Compliance
What is the difference between data drift and concept drift in Model Monitoring?
Data drift changes input distributions; concept drift changes the relationship between inputs and outputs. Both degrade performance but require different responses.
Category: Differences
What tools are commonly used for Model Monitoring?
Cloud-native tools (Vertex AI, SageMaker, Azure ML), open-source (Evidently, Prometheus, Grafana), and specialized platforms integrate with model registries and data catalogs.
Category: Tools
How should thresholds be set in Model Monitoring?
Use historical baselines, controlled experiments, cost-sensitive analysis, and iterative tuning to choose actionable, stable thresholds.
Category: Best practices