🎯L-BFGS

NEW

Also known as: Limited-memory BFGS

A limited-memory quasi-Newton method for large-scale optimization.

DESCRIPTION

L-BFGS is a quasi-Newton optimization algorithm that approximates second-order curvature using only a few recent gradient and parameter differences. It achieves near-Newton convergence rates with memory linear in the number of parameters. The method is well-suited for large-scale problems where storing or inverting the full Hessian is infeasible. It uses line search to ensure sufficient descent and stability. Variants such as L-BFGS-B support bound constraints. L-BFGS is widely implemented in scientific computing and machine learning libraries (e.g., SciPy, PyTorch) and often performs well on smooth, moderately ill-conditioned problems.

ALTERNATIVE DEFINITIONS

Wikipedia

Limited-memory BFGS (L-BFGS) is an optimization algorithm in the family of quasi-Newton methods that approximates the Broyden–Fletcher–Goldfarb–Shanno algorithm using a limited amount of computer memory.

Highlights quasi-Newton family and limited-memory approximation to BFGS.

Source

PyTorch

Implements L-BFGS algorithm, a quasi-Newton method.

Frames L-BFGS as a practical optimizer used to train models in deep learning frameworks.

Source

RELATED TERMS

Quasi-Newton Method Gradient Descent Newton’s Method Hessian Approximation Line Search Optimization Trust Region

COMPETING TERMS

Broyden–Fletcher–Goldfarb–Shanno Algorithm Conjugate Gradient Adam

KNOWLEDGE AREAS

Mathematics AI Data Science MLOps

TARGET AUDIENCE

Data Scientist AI Engineer MLOps Engineer Software Developer

EXTERNAL SOURCES

• Wikipedia - Limited-memory BFGS
• SciPy - SciPy L-BFGS-B optimizer reference
• PyTorch - PyTorch LBFGS optimizer docs
• libLBFGS - C library implementation of L-BFGS
• NYU Optimization Group - L-BFGS-B algorithm resources

TOP QUESTIONS

When should I use L-BFGS over Gradient Descent?

Use L-BFGS for smooth optimization problems where curvature information accelerates convergence. It usually outperforms first-order methods on well-conditioned, differentiable objectives with moderate noise.

Category: How-To

Why is L-BFGS memory-efficient?

It stores only a limited history of vector pairs from recent iterations, avoiding explicit Hessian storage. This yields quasi-Newton updates with memory scaling linearly in variables.

Category: Explanation

How does L-BFGS compare to BFGS?

L-BFGS approximates BFGS using limited history, enabling large-scale optimization. BFGS maintains a full dense Hessian approximation, which is memory-heavy for high-dimensional problems.

Category: Alternatives

What is a common mistake when using L-BFGS?

Applying it to highly non-smooth or noisy objectives can stall convergence. Ensure differentiability and use proper line search and tolerances to avoid premature termination.

Category: Mistakes

Does L-BFGS work for nonconvex objectives?

Yes, it is often effective for nonconvex problems but may converge to local minima. Initializations and line search settings influence outcomes significantly.

Category: Opinion

What is L-BFGS-B and when is it used?

L-BFGS-B is a bounded-constraint variant allowing simple box constraints. Use it when variables must remain within known lower and upper bounds.

Category: How-To

How does line search affect L-BFGS performance?

A robust line search ensures sufficient decrease and curvature conditions, stabilizing updates. Poor line search can cause slow progress or oscillation.

Category: Requirements

Is L-BFGS suitable for training deep neural networks?

It can be used but is less common than stochastic optimizers due to batch noise and cost of line searches. It may work well on smaller, smooth problems.

Category: Relevancy

What hyperparameters matter most for L-BFGS?

History size, tolerance criteria, and line search parameters are key. Reasonable defaults often work, but tuning can improve convergence speed and stability.

Category: Requirements