DataTerminology.comNEW
Also known as: Ridge regularization, Weight Decay
A regularization technique that adds the squared magnitude of coefficients as a penalty to reduce overfitting by discouraging large weights.
DESCRIPTION
L2 regularization, also known as ridge or Tikhonov regularization, penalizes the sum of squared parameter values. It shrinks coefficients toward zero without making most exactly zero, reducing variance while maintaining all featuresâ influence. In neural networks, L2 is commonly implemented as weight decay. L2 is effective when predictors are correlated and can improve numerical stability. It is often combined with L1 in Elastic Net to gain both stability and sparsity. Choosing the penalty strength via cross-validation is critical for optimal performance.
ALTERNATIVE DEFINITIONS
Wikipedia
Tikhonov regularization, also known as ridge regression or TikhonovâPhillips regularization, is a method of regularization of ill-posed problems by adding a penalty term, which is the square of the norm of the solution, to the objective function.
Connects L2 to stabilizing ill-posed problems via squared-norm penalty.
scikit-learn
Ridge regression addresses some of the problems of ordinary least squares by imposing a penalty on the size of coefficients.
Frames L2 as improving OLS by penalizing large coefficients.
RELATED TERMS
Regularization Ridge Regression Elastic Net Model Overfitting Bias-Variance Tradeoff Logistic Regression Linear Model Penalty Term Superclass
COMPETING TERMS
KNOWLEDGE AREAS
Mathematics AI Data Science MLOps
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Tikhonov regularization
- scikit-learn - Ridge regression documentation
- Stanford CS231n - Weight decay explanation for L2
- Goodfellow, Bengio, Courville - Deep learning regularization chapter
TOP QUESTIONS
When should I prefer L2 Regularization?
Prefer L2 when features are correlated and you want to reduce variance without eliminating features. It stabilizes estimates and often improves generalization.
Category: Application
How does L2 Regularization reduce overfitting?
By penalizing large weights, L2 discourages overly complex models, smoothing decision boundaries and reducing variance on unseen data.
Category: Mechanism
Is L2 Regularization the same as weight decay?
In many optimization settings, yes. Weight decay implements L2 by shrinking weights during updates, especially in neural networks.
Category: Misunderstanding
How do I tune the L2 penalty?
Use cross-validation to select lambda/alpha. Grid or randomized search over a logarithmic range is common practice.
Category: Requirements
What is a risk of over-regularization with L2?
Too much L2 increases bias and underfits data, degrading accuracy. Monitor validation performance to avoid this.
Category: Risks
How does L2 compare to L1 Regularization?
L2 spreads shrinkage across parameters without zeros; L1 induces sparsity by zeroing coefficients, useful for feature selection.
Category: Comparison
Does L2 help with multicollinearity?
Yes. Ridge shrinks correlated coefficients together, reducing variance and improving stability versus OLS.
Category: Benefits
Can L2 be used with classification?
Yes. Logistic regression and linear classifiers often include L2 penalties to regularize decision boundaries.
Category: Scope
What optimizers work well with L2?
SGD with weight decay and AdamW are popular choices in deep learning for effective L2 regularization.
Category: Best-Practices