DataTerminology.comNEW
A minimal text unit, such as a word or subword, treated as a single element for linguistic or computational processing.
DESCRIPTION
In NLP, tokens are atomic units created by tokenization for downstream tasks. Depending on the tokenizer, tokens may correspond to words, subwords, characters, or byte units. Subword tokenization (e.g., BPE, WordPiece) balances vocabulary size and out-of-vocabulary handling. Token definitions affect model vocabulary, sequence length, and performance. Proper handling requires attention to normalization, casing, and Unicode details. In parsing and compilers, tokens carry categorical meaning for syntax analysis.
ALTERNATIVE DEFINITIONS
Wikipedia
In computer science, a lexical token is a string with an assigned and thus identified meaning.
General CS definition emphasizing meaning assignment, applicable to NLP and parsing.
Stanford IR Book
A token is an instance of a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing.
Canonical IR definition used widely in NLP and search.
Google Developers
A token is a sequence of characters that is grouped together as a useful semantic unit for processing.
Developer-oriented gloss on the standard IR/NLP meaning.
RELATED TERMS
Tokenization Subword WordPiece Byte Pair Encoding SentencePiece Vocabulary N-gram Morpheme Tokenizer Unicode
COMPETING TERMS
Character Word
RELATIONS
- Narrower: Access Token ID Token Refresh Token
KNOWLEDGE AREAS
AI Data Science Programming Languages
TARGET AUDIENCE
AI Engineer Data Scientist Software Developer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Lexical token
- Hugging Face - Tokenizer library documentation
- Google SentencePiece - Unsupervised text tokenizer
- spaCy - Tokenization in spaCy
TOP QUESTIONS
Why do large language models use subword tokens?
Subwords balance vocabulary size and expressiveness, reducing out-of-vocabulary issues. They shorten sequences versus characters while handling rare words better than word-level tokenization.
Category: Relevancy
What is the impact of tokenization on model performance?
Tokenization influences sequence lengths, vocabulary size, and ability to represent morphology. Poor choices can increase compute or degrade accuracy.
Category: Requirements
How do tokens differ from words?
Tokens are processing units; they may be words, subwords, or characters. One word may map to multiple tokens or share tokens across words.
Category: Misunderstanding
What is a common mistake with tokens in preprocessing?
Mismatched tokenizers between training and inference cause degradation. Always keep the same tokenizer and normalization settings.
Category: Mistakes
How are tokens represented in Transformers?
Tokens map to integer IDs via a vocabulary, then to embeddings. Positional encodings add order information before attention layers.
Category: Example
Are tokens language-dependent?
Yes. Languages without clear word boundaries or rich morphology need tailored tokenization strategies like character or subword models.
Category: Requirements
How do byte-level tokens work?
They operate directly on bytes, ensuring full coverage of Unicode without reserved unknowns. This simplifies handling of rare symbols and emojis.
Category: Definition
What is vocabulary pruning and why does it matter?
Pruning removes infrequent tokens to shrink models and improve speed. Over-pruning can harm representation of rare words or domains.
Category: BestPractices
How do I choose between word, subword, and character tokens?
Consider language, domain, and model size. Subwords are a strong default; characters help with noisy text, words suit small, well-formed corpora.
Category: HowTo