DataTerminology.comNEW
Also known as: ERM
A learning principle that selects the hypothesis minimizing average loss over the training data.
DESCRIPTION
Empirical Risk Minimization (ERM) chooses the model that minimizes the empirical loss on observed samples. Under uniform convergence, minimizing empirical risk approximates minimizing true risk. ERM is the basis of many algorithms, including linear models and deep networks trained by gradient methods. Without capacity control, ERM can overfit, motivating regularization and SRM. ERM ties theory to practice by linking optimization of training loss to generalization guarantees.
ALTERNATIVE DEFINITIONS
Wikipedia
Empirical risk minimization (ERM) is a principle in statistical learning theory which suggests choosing the hypothesis that minimizes the empirical risk.
Highlights the principle and selection by empirical risk minimization.
Understanding Machine Learning (Shalev-Shwartz & Ben-David)
In ERM, the learner selects a hypothesis that minimizes the average loss over the training sample.
Explicitly states average-loss formulation central to ERM.
RELATED TERMS
Regularization Loss Function Training Error Generalization Error Probably Approximately Correct Learning VC Dimension Gradient Descent Model Overfitting Uniform Convergence
COMPETING TERMS
RELATIONS
- Broader: Risk Minimization
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Empirical risk minimization
- Stanford CS229 - Course notes on ERM and SRM
- MIT OCW - ERM in machine learning curriculum
TOP QUESTIONS
When does Empirical Risk Minimization generalize well?
When the hypothesis class has controlled capacity and uniform convergence holds, empirical risk approximates true risk, yielding good generalization.
Category: Requirements
How is Empirical Risk Minimization implemented?
Define a loss function and optimize average loss over the training set using methods like gradient descent, SGD, or convex solvers.
Category: Overview
What is a common pitfall of Empirical Risk Minimization?
Overfitting occurs if the hypothesis class is too expressive or regularization is absent; training loss can be low while test error is high.
Category: Mistakes
How does ERM relate to Maximum Likelihood Estimation?
With appropriate loss (e.g., negative log-likelihood), ERM coincides with maximum likelihood, linking statistical estimation and learning.
Category: Example
How can ERM be regularized?
Add complexity penalties (L1/L2, margin constraints) or early stopping to balance fit and capacity, improving generalization.
Category: Management
What risks arise when optimizing empirical risk alone?
Optimizing only training loss may exploit spurious patterns or noise, degrading robustness and fairness on real-world data.
Category: Risks
How does minibatch SGD affect ERM?
Stochastic estimates of gradients approximate ERM over time; noise can aid generalization and efficiency in large-scale settings.
Category: Opinion
Does ERM require i.i.d. data?
ERM analysis typically assumes i.i.d. samples; departures require specialized methods or bounds to retain guarantees.
Category: Requirements
How to monitor ERM performance during training?
Track validation and test loss alongside training loss, use early stopping, and assess robustness via cross-validation.
Category: Management