DataTerminology.comNEW
Also known as: k-NN
Non-parametric method that predicts labels or values from the majority or average of the k closest training points.
DESCRIPTION
K-nearest neighbors (KNN) classifies or regresses based on local neighborhoods in feature space. It is lazy learning: no explicit training beyond storing data. Performance hinges on distance metrics, feature scaling, and choice of k. Tree-based indexes (KD-tree, Ball tree) accelerate queries in moderate dimensions. KNN is simple and interpretable but struggles with high-dimensional data.
ALTERNATIVE DEFINITIONS
Wikipedia
In pattern recognition, the k-nearest neighbors algorithm (k-NN) is a non-parametric method used for classification and regression.
Introduces KNN as non-parametric with dual classification/regression use.
scikit-learn
The k-nearest neighbors algorithm classifies a sample by a majority vote of its neighbors, with the sample being assigned to the class most common among its k nearest neighbors.
Specifies majority-vote mechanism for classification.
RELATED TERMS
Dissimilarity Measure Classification Regression Analysis Feature Scaling Curse of Dimensionality KD-Tree Ball Tree Cross-Validation Weighted KNN
COMPETING TERMS
Support Vector Machine Decision Tree
RELATIONS
- Derivations: Approximate Nearest Neighbor
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer Data Analyst
EXTERNAL SOURCES
- Wikipedia - K-nearest neighbors algorithm
- scikit-learn - Neighbors algorithms and KNN usage
- IBM - Introductory guide to KNN
TOP QUESTIONS
When is K-nearest Neighbors a good choice?
When decision boundaries are locally smooth, features are scaled, and dimensionality is moderate. It excels with ample labeled data.
Category: Best practices
How do you choose k for K-nearest Neighbors?
Use cross-validation to balance bias and variance; odd k avoids ties in binary classification.
Category: How-to
What distance metrics are used in K-nearest Neighbors?
Common choices are Euclidean, Manhattan, Minkowski, cosine, or Hamming for categorical features.
Category: Overview
Why is feature scaling important for K-nearest Neighbors?
Unscaled features distort distances; standardization or normalization ensures fair neighbor comparisons.
Category: Mistakes
How does K-nearest Neighbors perform in high dimensions?
It suffers from the curse of dimensionality; dimensionality reduction or feature selection is advised.
Category: Risk
Can K-nearest Neighbors handle regression tasks?
Yes. It predicts the average (or weighted average) target of the k nearest neighbors.
Category: Misunderstanding
How can K-nearest Neighbors be sped up?
Use KD-trees, Ball trees, approximate nearest neighbor indices, and batch queries.
Category: Performance
What is weighted K-nearest Neighbors?
Neighbors closer to the query point receive higher weights, improving robustness in some settings.
Category: Definition
What are alternatives to K-nearest Neighbors?
Support Vector Machines, logistic regression, decision trees, and Naive Bayes are common alternatives.
Category: Alternatives