DataTerminology.comNEW
A phenomenon where test error decreases, spikes near interpolation, then decreases again as model capacity grows.
DESCRIPTION
Double descent generalizes the classical bias-variance tradeoff in modern regimes. As model capacity increases, test error initially falls, then rises near the interpolation threshold, and falls again with further scaling. It appears across neural networks and random feature models. Understanding this behavior guides model selection and regularization. Careful evaluation avoids overinterpreting intermediate spikes and leverages improved generalization at larger scales.
ALTERNATIVE DEFINITIONS
Wikipedia
Double descent is a phenomenon in which test error initially decreases with model complexity, then increases around the interpolation threshold, and decreases again as complexity grows.
Explicitly describes the two descent phases and interpolation spike.
Nakkiran et al.
We observe a double descent in test error as a function of model size, data size, and training time in modern ML systems.
Extends double descent beyond model size to data and time dimensions.
RELATED TERMS
Model Overfitting Interpolation Threshold Model Capacity Regularization Superclass Test Error Validation Curve Model Underfitting Model Complexity
COMPETING TERMS
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Double descent
- Nakkiran et al. - Empirical study of double descent
- Distill - Interactive article explaining double descent
TOP QUESTIONS
What is Double Descent?
It’s a generalization phenomenon where error decreases, spikes near interpolation, and decreases again with increasing model capacity.
Category: Definition
Why does Double Descent matter?
It challenges classical bias-variance intuition and informs decisions about scaling models and datasets.
Category: Relevancy
Where is Double Descent observed?
In deep neural networks, random features, and other modern ML settings across model size, data, and training time.
Category: Example
How do you mitigate the interpolation spike?
Use regularization, early stopping, or increase data and capacity to reach the second descent regime.
Category: Mitigation
How does Double Descent relate to the bias-variance tradeoff?
It extends the tradeoff, showing improved generalization at larger scales beyond the interpolation threshold.
Category: Distinction
What metrics reveal Double Descent?
Plot test error versus model capacity, data size, or training time to observe the characteristic curve.
Category: Measurement
What’s a misunderstanding about Double Descent?
That the spike indicates inevitable failure; increasing scale or regularization can reach improved generalization.
Category: Misunderstanding
Is Double Descent universal?
No, it depends on task, architecture, data, and training regimen, though it’s common in modern ML.
Category: Opinion
Where to learn more about Double Descent?
Read Nakkiran’s paper and Distill’s interactive article for intuition and evidence.
Category: GettingStarted