DataTerminology.comNEW
A plot comparing predicted probabilities to observed outcome frequencies to assess how well a classifier’s probabilities are calibrated.
DESCRIPTION
Calibration curves, also called reliability diagrams, visualize how predicted probabilities align with actual outcome frequencies across bins. A perfectly calibrated model lies close to the diagonal. Deviations reveal overconfidence or underconfidence, guiding recalibration methods like Platt scaling, isotonic regression, or temperature scaling. They complement metrics such as Brier score and Expected Calibration Error. In risk-sensitive domains, calibration is as important as discrimination (ROC/PR).
ALTERNATIVE DEFINITIONS
scikit-learn
Calibration curves, also referred to as reliability diagrams, compare how well the probabilistic predictions of a binary classifier are calibrated.
Explicitly ties curves to binary classifier probability calibration.
NIST/SEMATECH e-Handbook
A calibration curve is a general method for determining the concentration of a substance by comparison to a set of standard samples of known concentration.
Defines calibration curve in analytical measurement, illustrating the general concept of observed vs. expected.
RELATED TERMS
Reliability Diagram Expected Calibration Error Brier Score Platt Scaling Isotonic Regression Calibration Temperature Scaling Probability Forecasting
COMPETING TERMS
ROC Curve Precision-Recall Curve
KNOWLEDGE AREAS
AI Data Science Testing & Validation MLOps
TARGET AUDIENCE
Data Scientist MLOps Engineer AI Engineer BI Specialist
EXTERNAL SOURCES
- Wikipedia - Calibration (statistics)
- scikit-learn - User guide for probability calibration.
- Google Developers - MLCC lesson on calibration.
TOP QUESTIONS
Why is a Calibration Curve important for classifiers?
It reveals whether predicted probabilities reflect true outcome frequencies. Good calibration is crucial for risk-sensitive decisions, thresholding, and cost-sensitive optimization.
Category: Relevancy
How do I build a Calibration Curve for a model?
Bin predictions by probability, compute observed positive rate per bin, and plot observed vs. predicted. Repeat with cross-validation for robust estimates.
Category: Practice
What does it mean if the Calibration Curve lies above the diagonal?
The model is underconfident—observed frequencies exceed predicted probabilities. You may increase predicted probabilities or recalibrate to correct bias.
Category: Interpretation
How does a Calibration Curve differ from an ROC Curve?
Calibration curves assess probability calibration, while ROC focuses on discrimination (TPR vs. FPR). Both are complementary diagnostics.
Category: Differences
Which metrics complement a Calibration Curve?
Brier score, log loss, and Expected Calibration Error quantify calibration quality. Use them alongside visual inspection.
Category: Metrics
How do I fix a poor Calibration Curve?
Apply post-hoc calibration methods like Platt scaling, isotonic regression, or temperature scaling on validation data, then re-evaluate.
Category: Remediation
Should I calibrate probabilistic outputs for all models?
Calibrate when decisions depend on accurate probabilities. Tree ensembles and neural networks often need calibration; some models like logistic regression are often well calibrated.
Category: Opinion
Can I use a Calibration Curve for multi-class problems?
Yes, by one-vs-rest calibration per class or using top-label calibration; interpret carefully across classes.
Category: Practice
What data split should I use for Calibration Curves?
Use a validation or calibration holdout separate from training to prevent leakage. Cross-validation improves stability on small datasets.
Category: Requirements