DataTerminology.comNEW
Optimizing by iteratively updating one variable at a time.
DESCRIPTION
Coordinate descent minimizes an objective by successively optimizing along one coordinate (or block) while keeping others fixed. It is simple, often efficient for high-dimensional sparse problems, and powers solvers for lasso and logistic regression. Closed-form updates exist for some losses and regularizers, accelerating convergence. Variants include cyclic, randomized, and block coordinate descent. Convergence properties depend on convexity, Lipschitz continuity, and update rules. Proper scaling and feature normalization improve performance. It integrates well with proximal operators for L1 penalties.
ALTERNATIVE DEFINITIONS
Wikipedia
Coordinate descent is an optimization algorithm that successively minimizes along coordinate directions to find the minimum of a function.
Highlights successive minimization along single coordinates.
Optimization Online (Wright)
Coordinate descent methods optimize a multivariate function by successively minimizing with respect to a single coordinate direction.
Formal survey definition focusing on theoretical foundation.
RELATED TERMS
Block Coordinate Descent Lasso Ridge Regression Logistic Regression Convex Optimization Objective Function Convergence Analysis Optimization Solver
COMPETING TERMS
Gradient Descent Proximal Gradient
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer Software Developer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Coordinate descent
- scikit-learn - Lasso solver via coordinate descent
- Boyd (Stanford) - Coordinate descent lecture notes
- Berkeley EECS - Class notes on coordinate descent
TOP QUESTIONS
What is Coordinate Descent?
It is an optimization algorithm that updates one variable at a time while holding others fixed, iteratively moving toward a minimum.
Category: Definition
When is Coordinate Descent preferred?
For high-dimensional, sparse problems with separable structures, where coordinate-wise updates are computationally cheap and effective.
Category: Opinion
What are variants of Coordinate Descent?
Cyclic, randomized, and block coordinate descent, often combined with proximal updates for regularized objectives.
Category: Overview
What common mistakes occur with Coordinate Descent?
Poor feature scaling, inappropriate coordinate ordering, and ignoring convergence criteria can slow or destabilize optimization.
Category: Mistakes
How does Coordinate Descent compare to Gradient Descent?
Coordinate Descent performs 1D optimizations per step, while Gradient Descent uses full gradient steps; trade-offs hinge on problem structure and sparsity.
Category: Alternatives
How to ensure convergence in Coordinate Descent?
Use convex objectives, Lipschitz conditions, suitable step rules, and randomized coordinate selection to avoid pathological cycles.
Category: HowTo
What tasks commonly use Coordinate Descent?
Lasso, elastic net, and logistic regression often rely on coordinate-wise or block updates for efficiency.
Category: Overview
What are risks of using Coordinate Descent?
In nonconvex settings, it may converge to poor local minima or stall; careful initialization and regularization help.
Category: Risks
How to implement Coordinate Descent efficiently?
Exploit sparsity, precompute sufficient statistics, and use warm starts; vectorize or parallelize block updates where possible.
Category: HowTo