DataTerminology.comNEW
Methods approximating second-order information for efficient optimization.
DESCRIPTION
Quasi-Newton methods update an approximation to the Hessian or its inverse to guide search directions without computing exact second derivatives. Algorithms like BFGS and L-BFGS offer fast convergence on smooth problems and scale to large parameter spaces. They use line searches and curvature conditions to ensure stability. In ML, they optimize logistic regression and other convex objectives effectively. Limited-memory variants store compact representations for high-dimensional models. Proper termination and safeguarding address numerical issues. They bridge gradient methods and full Newton’s method.
ALTERNATIVE DEFINITIONS
Wikipedia
In numerical optimization, quasi-Newton methods are a class of methods used to find local maxima and minima of functions by using gradient information to build up an approximation to the Hessian matrix.
Defines the use of gradient-based Hessian approximations for optimization.
SciPy
BFGS is a quasi-Newton method for unconstrained minimization.
Identifies BFGS as a specific and widely used quasi-Newton algorithm.
RELATED TERMS
Broyden–Fletcher–Goldfarb–Shanno Algorithm L-BFGS Hessian Approximation Line Search Gradient Descent Conjugate Gradient Optimization Solver Curvature Unconstrained Optimization
COMPETING TERMS
Newton’s Method Stochastic Gradient Descent
RELATIONS
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer Software Developer
EXTERNAL SOURCES
- Wikipedia - Quasi-Newton method
- SciPy - BFGS optimizer documentation
- UBC CS - Lecture notes on quasi-Newton methods
- Stanford - Notes on BFGS and L-BFGS
TOP QUESTIONS
What is a Quasi-Newton Method?
It approximates second-order curvature (Hessian) using gradient information to produce efficient search directions without computing exact Hessians.
Category: Definition
Which algorithms are Quasi-Newton Methods?
BFGS and L-BFGS are the most widely used, with limited-memory variants suitable for high-dimensional problems.
Category: Overview
When to prefer Quasi-Newton over Gradient Descent?
When smooth objectives require faster convergence and better conditioning, especially in convex problems like logistic regression.
Category: Opinion
What are common mistakes with Quasi-Newton Methods?
Incorrect line search, poor initialization, or violating curvature conditions can break update stability or slow convergence.
Category: Mistakes
What risks exist using Quasi-Newton Methods?
In nonconvex settings, they may converge to saddle points; ill-conditioned problems can degrade performance without preconditioning.
Category: Risks
How do L-BFGS updates remain memory-efficient?
They store limited pairs of gradients and steps, reconstructing curvature information implicitly, reducing memory from O(n^2) to O(mn).
Category: Example
How to configure line search for Quasi-Newton Methods?
Use Wolfe or strong Wolfe conditions to ensure sufficient decrease and curvature, improving stability and convergence.
Category: HowTo
What tasks benefit from Quasi-Newton Methods?
Convex ML models like logistic regression, and smooth objectives in scientific computing, benefit from rapid convergence.
Category: Overview
What is the difference between Newton and Quasi-Newton?
Newton uses exact Hessians; Quasi-Newton uses approximations, trading accuracy for efficiency and robustness.
Category: Alternatives