DataTerminology.comNEW
A bandit-based resource allocation strategy that accelerates hyperparameter search via aggressive early stopping and successive halving.
DESCRIPTION
Hyperband frames hyperparameter tuning as a pure-exploration bandit problem with adaptive resource allocation. It evaluates many configurations with small budgets, then allocates more resources to the most promising ones. Building on Successive Halving, Hyperband chooses brackets and budgets to balance exploration and exploitation. It offers strong anytime performance and simplicity. It is widely implemented in Ray Tune, Optuna, and other HPO frameworks. ASHA extends it to asynchronous settings for distributed systems.
ALTERNATIVE DEFINITIONS
arXiv
We propose Hyperband, a simple and efficient method for hyperparameter optimization that uses adaptive early-stopping to allocate resources to promising configurations.
Focuses on adaptive early-stopping as core mechanism.
Ray
HyperBand is a bandit-based early-stopping algorithm for hyperparameter optimization.
Concise practical definition within an HPO framework.
RELATED TERMS
Grid Search Successive Halving Asynchronous Successive Halving Algorithm Hyperparameter Tuning Early Stopping Ray Tune Optuna Bandit Algorithms
COMPETING TERMS
Bayesian Optimization Random Search
KNOWLEDGE AREAS
MLOps AI Data Science Testing & Validation
TARGET AUDIENCE
MLOps Engineer Data Scientist AI Engineer
EXTERNAL SOURCES
- Wikipedia - Hyperparameter optimization#Bandit-based methods
- Ray - Ray Tune HyperBand tutorial
- Optuna - Optuna Hyperband pruner docs
TOP QUESTIONS
What is Hyperband and why is it fast?
Hyperband rapidly narrows the search by stopping poor configurations early and reallocating budgets to promising ones. This aggressive pruning yields strong results with fewer resources.
Category: Definition
How does Hyperband relate to Successive Halving?
Hyperband generalizes Successive Halving by selecting different brackets and budgets, balancing exploration and exploitation automatically.
Category: Concepts
When should I prefer Hyperband over Bayesian Optimization?
Choose Hyperband for high-dimensional or noisy searches with cheap-to-evaluate budgets and when parallel resources allow evaluating many configurations quickly.
Category: Decision
What is ASHA and how does it compare to Hyperband?
ASHA is an asynchronous extension enabling distributed, lock-free scheduling, often outperforming synchronous Hyperband in cluster environments.
Category: Alternatives
What are common mistakes when using Hyperband?
Using budgets that are too small to assess performance, ignoring minimum epochs, or misaligned fidelity metrics can lead to premature pruning of good candidates.
Category: Mistakes
How do I choose the resource to allocate in Hyperband?
Use a monotonic fidelity like epochs, steps, data subset size, or wall-clock time that correlates with performance improvement.
Category: HowTo
Does Hyperband guarantee the best configuration?
No. It is an anytime heuristic that trades completeness for speed, but it often finds near-optimal configurations efficiently.
Category: Misunderstanding
What metrics should I monitor during Hyperband runs?
Track validation loss/accuracy, training time, resource utilization, and pruning events to ensure meaningful comparisons across budgets.
Category: Management
Is Hyperband suitable for large models like LLMs?
It can be used with scalable fidelity (e.g., steps or tokens), but the cost and variance may favor more informed strategies or warm-starting.
Category: UseCases