DataTerminology.comNEW
Also known as: TF-IDF
A weighting scheme that scores term importance by combining within-document frequency with corpus-wide rarity.
DESCRIPTION
TF-IDF assigns higher weights to terms frequent in a document but rare across the corpus. It underpins vector-space retrieval, document similarity, and classic text classification. Variants adjust term frequency (e.g., sublinear, log) and IDF smoothing to handle edge cases. Resulting vectors are sparse and high-dimensional, often normalized (e.g., L2). Despite its simplicity, TF-IDF remains competitive for keyword search and baselines. Limitations include ignoring word order and semantics. Modern systems may combine TF-IDF with embeddings or BM25 for better ranking.
ALTERNATIVE DEFINITIONS
Wikipedia
In information retrieval, tf–idf, short for term frequency–inverse document frequency, is a numerical statistic intended to reflect how important a word is to a document in a collection or corpus.
Emphasizes importance weighting across document and corpus.
scikit-learn
Tf-idf stands for term-frequency times inverse document-frequency and is a popular scheme for weighting a term in a document.
Concise definition focused on feature weighting for modeling.
Stanford IR Book
The tf–idf weight (term frequency–inverse document frequency) is a weight often used in information retrieval and text mining.
Defines tf–idf in the canonical IR textbook context.
RELATED TERMS
Vector Space Model Bag-of-Words Cosine Similarity Document-Term Matrix Stemming Stopwords Feature Engineering Information Retrieval Text Mining Lemmatization Count Vectorizer Feature Extraction
COMPETING TERMS
BM25 Word Embedding
RELATIONS
- Broader: Frequency
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer Data Engineer Analytics Engineer Software Developer
EXTERNAL SOURCES
- Wikipedia - Tf–idf
- scikit-learn - Text feature extraction user guide
- Stanford IR Book - IR textbook on weighting
- scikit-learn - TF-IDF vectorizer API
- Stanford IR Book - Textbook chapter on tf–idf
- Apache Lucene - Scoring with tf and idf
- Google Developers - Glossary definition of tf-idf
TOP QUESTIONS
What problem does TF-IDF solve?
It balances term frequency with corpus-wide rarity to highlight discriminative terms and downweight common words like stop words.
Category: Overview
How is TF-IDF computed?
Compute term frequency per document, multiply by inverse document frequency, often applying log scaling and document length normalization.
Category: Mechanics
How does TF-IDF compare to BM25?
BM25 generally outperforms TF-IDF by modeling term saturation and document length more robustly using probabilistic foundations.
Category: Comparison
Is TF-IDF still relevant with embeddings?
Yes; it’s fast, interpretable, and effective for sparse features, baselines, and exact keyword relevance tasks.
Category: Relevancy
What are common mistakes with TF-IDF?
Ignoring preprocessing like stemming and stop-word removal, or failing to normalize by document length, weakens results.
Category: Mistakes
How is TF-IDF used in search?
Documents and queries are represented as TF-IDF vectors; cosine similarity ranks documents by weighted term overlap.
Category: Application
What is a hot topic around TF-IDF?
Hybrid retrieval uses TF-IDF/BM25 for candidate generation then reranks with transformer models for improved relevance.
Category: Topic
Where should I learn TF-IDF basics?
Start with the scikit-learn feature extraction docs and the Stanford IR textbook for foundations and math.
Category: Learning
What are variations of TF-IDF?
Variants include log-TF, sublinear TF, different IDF definitions, and normalization strategies to improve robustness.
Category: Variants