DataTerminology.comNEW
Also known as: BFGS
A quasi-Newton optimization algorithm that iteratively approximates the inverse Hessian to achieve fast local convergence.
DESCRIPTION
BFGS updates an approximation to the inverse Hessian using gradient differences and step vectors, enabling curvature-aware steps without second derivatives. With line search, it achieves superlinear convergence near optima for smooth problems. It is effective for medium-scale, smooth, unconstrained optimization. L-BFGS extends it to large problems by storing limited vectors. Performance depends on function smoothness and well-conditioned scaling.
ALTERNATIVE DEFINITIONS
Wikipedia
The BFGS method is an iterative method for solving unconstrained nonlinear optimization problems, updating an approximation to the Hessian matrix.
Defines iterative Hessian approximation for unconstrained problems.
SciPy
Minimize a function using the BFGS algorithm, a quasi-Newton method that uses function values and gradients to approximate the Hessian.
Developer-oriented API description with method characterization.
Wolfram MathWorld
BFGS is a quasi-Newton method that updates an approximate inverse Hessian to determine search directions in nonlinear optimization.
Concise mathematical characterization emphasizing inverse Hessian updates.
RELATED TERMS
Quasi-Newton Method Hessian Line Search Gradient Trust Region Optimization Conditioning
COMPETING TERMS
L-BFGS Conjugate Gradient Newton’s Method
RELATIONS
- Broader: Algorithm Quasi-Newton Method
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Broyden–Fletcher–Goldfarb–Shanno algorithm
- Boyd & Vandenberghe - Convex optimization reference
- Northwestern Optimization Wiki - Overview of quasi-Newton methods
- PyTorch - L-BFGS optimizer API
TOP QUESTIONS
When should I use BFGS over SGD?
Use BFGS for smooth, medium-scale problems where gradient evaluations are affordable and fast local convergence is desired.
Category: Requirements
What are BFGS’s limitations?
It struggles with noisy gradients, large-scale problems due to memory/time, and non-smooth objectives.
Category: Risks
How does L-BFGS differ from BFGS?
L-BFGS stores a limited number of vectors to approximate the inverse Hessian without forming a full matrix, enabling larger problems.
Category: Alternatives
Does BFGS guarantee global optimality?
No. It is a local optimizer and may converge to local minima for non-convex problems.
Category: Misunderstanding
What line search is recommended for BFGS?
Wolfe or strong Wolfe conditions are commonly used for robust step selection.
Category: Requirements
Is BFGS suitable for deep neural networks?
Typically no; stochastic methods scale better. BFGS can be used for small networks or fine-tuning convex subproblems.
Category: Opinion
How do I initialize BFGS?
Start with the identity for the inverse Hessian approximation and use scaled variants to improve conditioning.
Category: Management
What metrics track BFGS convergence?
Monitor gradient norm, objective value decrease, and step sizes across iterations.
Category: Testing
What is a hot topic around BFGS?
Hessian-free and hybrid second-order methods that scale to larger models are active research.
Category: Topic