DataTerminology.comNEW
A probabilistic ranking function scoring documents by term frequency, inverse document frequency, and length normalization.
DESCRIPTION
BM25 (Okapi BM25) is a classic retrieval function derived from the probabilistic relevance framework. It ranks documents based on term frequency saturation, inverse document frequency, and document length normalization with tunable parameters k1 and b. BM25 is robust, simple, and widely used as a baseline in search systems like Lucene and Elasticsearch. Variants such as BM25+ and BM25L adjust length normalization and low-frequency handling. Effective use depends on tokenization, stemming, stopword handling, and parameter tuning via relevance judgments.
ALTERNATIVE DEFINITIONS
Wikipedia
Okapi BM25 is a ranking function used by search engines to estimate the relevance of documents to a given search query.
Concise statement focusing on ranking and relevance estimation.
Elasticsearch
BM25 is a ranking function that uses term frequency, inverse document frequency, and document length normalization.
Highlights the three core components used by BM25 in practice.
Apache Lucene
Implements the Okapi BM25 ranking function.
Indicates direct implementation within a widely used search library.
RELATED TERMS
Okapi BM25 Probabilistic Information Retrieval Document Length Normalization Lucene Elasticsearch Relevance Feedback BM25L
COMPETING TERMS
Term Frequency–Inverse Document Frequency Query Likelihood Model
RELATIONS
- Equivalent to: Okapi BM25
- Narrower: Okapi BM25
KNOWLEDGE AREAS
AI Data Science Platform & Tooling
TARGET AUDIENCE
AI Engineer Data Scientist Software Developer Data Engineer Solution Architect
EXTERNAL SOURCES
- Wikipedia - Okapi BM25
- Apache Lucene - BM25Similarity reference implementation
- Elasticsearch - Similarity options including BM25
- University of Waterloo - Foundational BM25 paper and theory
TOP QUESTIONS
What problem does BM25 solve?
BM25 scores and ranks documents for a query, estimating relevance based on term statistics and document length.
Category: Definition
How do BM25 parameters k1 and b affect ranking?
k1 controls term frequency saturation; b controls length normalization. Tuning them adjusts how heavily longer documents and repeated terms are weighted.
Category: Parameters
Is BM25 still relevant with neural search?
Yes. It’s a strong baseline and often combined with embeddings in hybrid retrieval systems for speed and precision.
Category: Relevancy
When is BM25 preferred over TF-IDF?
BM25 generally outperforms TF-IDF due to TF saturation and length normalization, offering better retrieval effectiveness.
Category: Comparison
What are common mistakes using BM25?
Poor tokenization, inconsistent stemming, and not tuning k1/b can degrade performance. Evaluate on representative queries and judgments.
Category: Mistakes
How do I tune BM25?
Use a validation set with relevance judgments, grid search k1 and b, and iterate with offline metrics like MAP and NDCG.
Category: BestPractices
What are BM25 variants?
BM25+, BM25L, and BM25Adpt modify saturation and length normalization to address edge cases and long document biases.
Category: Concepts
Does BM25 work for multi-lingual search?
Yes, but effectiveness depends on language-specific tokenization, morphology handling, and stopword lists.
Category: Usage
Can BM25 be used for passage retrieval?
Yes. Apply BM25 to passages or sections to improve matching granularity and answer extraction.
Category: Application