DataTerminology.comNEW
Probabilistic global optimization for expensive black-box functions using a surrogate model and acquisition strategy.
DESCRIPTION
Bayesian optimization builds a probabilistic surrogate of an expensive objective (e.g., model validation score) and selects new evaluations via an acquisition function balancing exploration and exploitation. Gaussian processes are common surrogates, though random forests and neural networks are used. It excels when evaluations are costly or noisy and gradients are unavailable. Applications include hyperparameter tuning and experimental design. Extensions handle constraints, multi-objective, and multi-fidelity scenarios.
ALTERNATIVE DEFINITIONS
Wikipedia
Bayesian optimization is a strategy for the global optimization of objective functions that are expensive to evaluate.
Emphasizes global optimization of expensive functions.
Brochu, Cora, de Freitas (2010)
Bayesian optimization is a sequential design strategy for global optimization of black-box functions that does not require derivatives.
Highlights sequential design and derivative-free property.
scikit-optimize
Bayesian optimization uses a surrogate model to approximate the objective and an acquisition function to decide where to sample next.
Focuses on surrogate and acquisition function mechanics.
RELATED TERMS
Hyperparameter Tuning Gaussian Processes Surrogate Model Acquisition Function Expected Improvement Thompson Sampling Black-box Optimization Multi-fidelity Optimization
COMPETING TERMS
Random Search Grid Search Hyperband/BOHB
RELATIONS
- Broader: Optimization
- Narrower: BoTorch
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Bayesian optimization
- arXiv - Tutorial on Bayesian optimization
- Optuna - Hyperparameter optimization framework
- Google Cloud - Black-box optimization service
TOP QUESTIONS
When is Bayesian Optimization preferred?
When objective evaluations are expensive, noisy, or derivative-free, and you have a limited evaluation budget.
Category: Decision
What surrogate models are used in Bayesian Optimization?
Gaussian processes are common; random forests and neural networks are alternatives for high-dimensional or large datasets.
Category: Technique
How does the acquisition function work?
It quantifies utility of sampling locations, balancing exploration and exploitation (e.g., Expected Improvement, UCB).
Category: Concept
Can Bayesian Optimization handle constraints?
Yes, by modeling constraints probabilistically or penalizing infeasible regions in the acquisition function.
Category: Practice
Is Bayesian Optimization parallelizable?
Batch acquisitions and fantasizing enable parallel evaluations, but introduce trade-offs in sample efficiency.
Category: Tradeoff
How sensitive is it to hyperparameters?
Surrogate kernel choices and acquisition parameters matter; cross-validation and priors improve robustness.
Category: Tuning
Does it scale to high dimensions?
Performance degrades with dimension; embeddings, additive kernels, or bandit methods can help.
Category: Risk
What are alternatives to Bayesian Optimization?
Random search, grid search, Hyperband/BOHB, evolutionary strategies, and gradient-based tuning when differentiable.
Category: Alternatives
How does noise affect Bayesian Optimization?
Noise can be modeled in the surrogate; repeated evaluations or robust acquisition functions mitigate its impact.
Category: Analysis