DataTerminology.comNEW
Also known as: ASHA
An asynchronous, bandit-based hyperparameter optimization method that promotes promising trials and prunes poor ones to save compute.
DESCRIPTION
ASHA extends Successive Halving by removing synchronization barriers, allowing workers to start, promote, or stop trials independently. It allocates increasing resources to top performers across rungs while aggressively terminating underperformers early. This improves hardware utilization in distributed environments and reduces wall-clock time. ASHA is widely supported in open-source tooling like Ray Tune and Optuna. It is effective for large search spaces and noisy evaluations, particularly on clusters.
ALTERNATIVE DEFINITIONS
Ray
Asynchronous Successive Halving (ASHA) is a bandit-based early stopping algorithm for hyperparameter optimization that aggressively terminates poor-performing trials.
Emphasizes early stopping and bandit framing with aggressive pruning for efficiency.
arXiv
We propose ASHA, a simple and robust method for asynchronous parallel hyperparameter optimization that aggressively terminates poorly-performing trials.
Introduces the algorithm with focus on asynchrony and robustness in parallel tuning.
RELATED TERMS
Successive Halving Early Stopping Random Search Grid Search Population Based Training Hyperparameter Tuning Ray Tune Optuna
COMPETING TERMS
Hyperband Bayesian Optimization
RELATIONS
- Broader: Algorithm Hyperparameter Tuning
KNOWLEDGE AREAS
AI MLOps Data Science Testing & Validation
TARGET AUDIENCE
MLOps Engineer Data Scientist AI Engineer AI Product Manager
EXTERNAL SOURCES
- Wikipedia - Hyperparameter optimization
- Ray - Ray Tune docs with ASHA scheduler
- Optuna - Pruner reference related to ASHA behavior
- arXiv - Original Successive Halving paper
TOP QUESTIONS
When should I use ASHA for hyperparameter tuning?
Use ASHA when evaluations are expensive, parallel resources are available, and you want to minimize synchronization overhead and quickly terminate weak trials.
Category: Guidance
How does ASHA differ from Hyperband?
ASHA removes Hyperband’s synchronization barriers, enabling continuous promotion and pruning, which often yields better hardware utilization and faster time-to-result.
Category: Comparison
What is the biggest risk when using ASHA?
Over-aggressive pruning can discard trials that would perform well given more resources. Calibrate rungs and promotion thresholds to mitigate premature termination.
Category: Risks
What hyperparameters of ASHA matter most?
Key settings include the minimum resource per trial, reduction factor (η), number of rungs, and promotion thresholds that govern pruning aggressiveness.
Category: Requirements
Is ASHA suitable for non-monotonic learning curves?
It can be, but non-monotonic curves increase the risk of premature pruning. Consider gentler thresholds or complementary methods like Hyperband or BO for safety.
Category: Opinion
What tooling supports ASHA?
Ray Tune, Optuna, and other HPO frameworks include ASHA or compatible early-stopping pruners, making it easy to adopt in distributed settings.
Category: Tools
How does ASHA save compute?
It allocates more resources to promising trials while terminating poor ones early, reducing wasted compute on unpromising configurations.
Category: Concept
Can ASHA handle large search spaces?
Yes. Its bandit-based pruning scales well, especially when combined with random or quasi-random sampling across vast hyperparameter spaces.
Category: Relevancy
How do I start learning ASHA?
Begin with Ray Tune tutorials and the ASHA paper, then run small experiments to understand rung design and pruning behavior.
Category: GettingStarted