DataTerminology.comNEW
Selecting a subset of relevant features to improve model generalization, efficiency, and interpretability.
DESCRIPTION
Feature selection reduces input dimensionality by retaining informative variables and discarding redundant or noisy ones. Methods include filters (statistical tests), wrappers (RFE, stepwise), and embedded approaches (LASSO, tree-based importance). Benefits include improved generalization, faster training, and simpler explanations. It’s particularly helpful in high-dimensional, low-sample-size problems. Validation via cross-validation and stability analysis ensures robustness.
ALTERNATIVE DEFINITIONS
Wikipedia
Feature selection is the process of selecting a subset of relevant features for use in model construction.
Defines relevance and selection for model building.
scikit-learn
Feature selection is the process of selecting a subset of relevant features for use in model construction, improving generalization by reducing overfitting.
Adds generalization and overfitting reduction.
IBM
Feature selection is a process used to identify and select the most important features from a dataset to build effective predictive models.
Vendor-oriented emphasis on importance and effectiveness.
RELATED TERMS
Feature Engineering Lasso Recursive Feature Elimination Mutual Information Filter Methods Wrapper Methods Embedded Methods Regularization
COMPETING TERMS
Feature Reduction Feature Extraction
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer Data Analyst
EXTERNAL SOURCES
- Wikipedia - Feature selection
- scikit-learn - Comprehensive selection methods
- Google Developers - Guidance on selecting features
- Interpretable Machine Learning - Interpretability and selection chapter
TOP QUESTIONS
Why perform Feature Selection?
To reduce overfitting, improve generalization, speed up training, and simplify interpretation. It’s critical in high-dimensional datasets with limited samples.
Category: Relevancy
What are the main types of Feature Selection methods?
Filter (statistical scores), wrapper (search with model), and embedded (built-in, e.g., LASSO) methods. Hybrid strategies are common in practice.
Category: Overview
How do I avoid leakage during Feature Selection?
Perform selection within each cross-validation fold to prevent information bleeding from validation to training data.
Category: Risk
How many features should I select?
There’s no universal number. Use validation performance, domain knowledge, and stability across folds to decide.
Category: Opinion
Can tree-based importance be trusted for Feature Selection?
It’s useful but can be biased toward high-cardinality or noisy features. Use permutation importance and validate with ablation.
Category: Misunderstanding
What is stability selection?
A technique that assesses how consistently features are selected across subsamples, improving robustness of selection decisions.
Category: Testing
How does Feature Selection affect interpretability?
Fewer, more relevant features often improve interpretability. However, some embedded methods may choose non-intuitive subsets.
Category: Opinion
When to prefer Feature Extraction over Selection?
Extraction may outperform selection when features are highly correlated or unstructured; selection is preferable for interpretability and governance.
Category: Alternatives
How to operationalize Feature Selection?
Automate selection within training pipelines, store selected feature lists with model versions, and monitor feature availability and drift.
Category: Management