DataTerminology.comNEW
Also known as: Local Minimum, Local maximum
A solution that is optimal within a small neighborhood but not necessarily the best overall solution.
DESCRIPTION
Local optima arise in nonconvex optimization where many peaks and valleys exist. Gradient methods can become trapped near local minima or maxima, or stall at saddle points. In high-dimensional deep learning, saddle points are often more problematic than true bad local minima. Techniques like random restarts, momentum, learning-rate schedules, and stochasticity help escape poor basins. Understanding local optima is key to interpreting training dynamics and generalization.
ALTERNATIVE DEFINITIONS
Wikipedia
In applied mathematics, a local optimum of an optimization problem is a solution that is optimal (either maximal or minimal) within a neighboring set of candidate solutions.
Formalizes optimality within a neighborhood rather than globally.
Wolfram MathWorld
A local minimum of a function is a point where the function value is smaller than at all nearby points.
Gives a concise mathematical characterization for minima.
RELATED TERMS
Saddle Point Gradient Descent Nonconvex Optimization Convex Optimization Loss Landscape Annealing Initialization Momentum Hessian
COMPETING TERMS
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Local optimum
- UBC CPSC 540 - Advanced optimization course notes
- Goodfellow, Bengio, Courville - Optimization and loss landscapes
- Stanford CS231n - Optimization issues in deep learning
TOP QUESTIONS
Why do Local Optima matter in training models?
They can trap optimization, leading to suboptimal performance. Recognizing and mitigating them improves convergence and generalization.
Category: Relevancy
Are Local Optima a big problem in deep learning?
Often less than feared; saddle points and plateaus are more common. Stochastic gradients and modern heuristics help navigate complex landscapes.
Category: Opinion
How can I escape poor Local Optima?
Use random restarts, momentum, adaptive optimizers, learning-rate schedules, and noise-injection techniques.
Category: Best-Practices
What is the difference between Local and Global Optima?
Local optima are best only nearby; global optima are the best overall across the feasible region.
Category: Comparison
How does initialization affect Local Optima?
Initial weights determine the basin reached by gradient methods. Good initializations improve convergence and reduce trapping.
Category: Requirements
Can regularization influence Local Optima?
Yes. Regularization reshapes the landscape, potentially smoothing it and favoring wider, more generalizable minima.
Category: Mechanism
How do we diagnose being stuck at a Local Optimum?
Flat validation metrics, low gradient norms, and sensitivity to restarts suggest plateau or local basin issues.
Category: Evaluation
Is convex optimization free of Local Optima?
Yes for minima: any local minimum is global in convex problems, simplifying optimization.
Category: Theory
What is a hot topic related to Local Optima?
Understanding generalization via flat versus sharp minima and implicit bias of optimizers is an active research area.
Category: Topic