DataTerminology.comNEW
A configuration value set before training that controls the learning algorithm’s behavior or model capacity.
DESCRIPTION
Hyperparameters govern model complexity, optimization dynamics, and regularization. Examples include learning rate, number of layers, batch size, and penalty strengths. Unlike learned parameters, they are not fit from data but chosen via tuning strategies. Proper hyperparameter selection improves generalization and training stability. Automated search methods, priors, and early stopping reduce manual effort and overfitting risk. Documenting chosen values ensures reproducibility.
ALTERNATIVE DEFINITIONS
Wikipedia
In machine learning, a hyperparameter is a parameter whose value is used to control the learning process.
Highlights control over the learning process rather than being learned from data.
scikit-learn
Hyper-parameters are parameters that are not directly learned within estimators.
Contrasts hyperparameters with learned model parameters in estimators.
RELATED TERMS
Learning Rate Batch Size Regularization Epoch Architecture Search Early Stopping Grid Search Random Search Bayesian Optimization
COMPETING TERMS
Model Parameters
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Hyperparameter (machine learning)
- scikit-learn - Guidance on hyperparameter tuning
- Google Developers ML Glossary - Glossary entry on hyperparameters
- Optuna - Framework for hyperparameter optimization
TOP QUESTIONS
How do hyperparameters differ from model parameters?
Model parameters are learned from data (e.g., weights), while hyperparameters are set before training to control learning dynamics or capacity.
Category: Misunderstanding
Which hyperparameters matter most in deep learning?
Learning rate, batch size, optimizer choices, weight decay, and architectural depth often dominate performance and stability.
Category: Opinion
What is a good starting point for learning rate?
Use learning rate ranges or schedulers (cosine, one-cycle) to explore. Start conservatively, then tune using validation metrics.
Category: Best Practice
How do I avoid overfitting when tuning hyperparameters?
Use cross-validation, early stopping, and separate validation/holdout sets. Limit search complexity and track experiments.
Category: Risk
Can hyperparameters be learned automatically?
Yes, via automated search methods or differentiable hyperparameter optimization, though the latter is more advanced and niche.
Category: Technique
Should I tune all hyperparameters at once?
Prioritize the most sensitive ones (learning rate, regularization) and iterate. Use hierarchical or sequential search strategies.
Category: Best Practice
How do I document hyperparameters for reproducibility?
Record values, search spaces, seeds, data versions, and environment details in experiment tracking systems.
Category: Management
What is the interaction between batch size and learning rate?
Larger batches often permit higher learning rates; scaling rules can maintain training dynamics.
Category: Concept
How do constraints help in hyperparameter settings?
Constraints (bounds, conditional dependencies) shrink the search space and prevent invalid configurations.
Category: Requirements