🔤Vocabulary

NEW

The set of tokens recognized by a tokenizer and language model, mapped to unique identifiers.

DESCRIPTION

In NLP, a vocabulary defines the discrete set of tokens a model can process, each mapped to an integer ID. Vocabularies can be word-level, subword-level, or byte-level to balance coverage and efficiency. Subword schemes like WordPiece or Byte Pair Encoding reduce out-of-vocabulary issues while controlling sequence length. The vocabulary interacts with tokenizers, embeddings, and model architectures. Managing vocabulary size affects memory, speed, and generalization. Domain-specific vocabularies improve performance on specialized text.

ALTERNATIVE DEFINITIONS

Hugging Face

A vocabulary is the set of tokens known to a tokenizer and used by models.

Directly ties vocabulary to tokenizers and model usage.

Source

spaCy

The Vocab object stores all lexical entries, strings and word hashes used by the model.

Focuses on lexical storage and model integration in spaCy.

Source

Google Research (WordPiece)

WordPiece uses a fixed-size vocabulary of subword units to efficiently represent text.

Highlights subword units and fixed-size vocabulary efficiency.

Source

RELATED TERMS

Tokenization Token Subword Vocabulary Byte Pair Encoding WordPiece SentencePiece Embeddings Out-of-Vocabulary Tokenizer Vocabulary Size

COMPETING TERMS

Character-level Modeling Byte-level Vocabulary

RELATIONS

• Narrower: Controlled Vocabulary RDF Vocabulary

KNOWLEDGE AREAS

AI Data Science Semantic & Knowledge Modeling

TARGET AUDIENCE

AI Engineer Data Scientist Software Developer

EXTERNAL SOURCES

• Wikipedia - Vocabulary
• Hugging Face - Tokenizer-vocabulary summary
• Google SentencePiece - Subword vocabulary toolkit
• TensorFlow - Subword tokenization guide

TOP QUESTIONS

Why does Vocabulary size matter in language models?

It affects memory, compute, and sequence lengths; larger vocabularies reduce token count but increase embedding size.

Category: Tradeoff

What is an out-of-vocabulary token in Vocabulary?

A token not present in the vocabulary; subword and byte-level approaches mitigate OOVs.

Category: Concept

How is a Vocabulary created?

Often by training a tokenizer on corpora using algorithms like WordPiece or BPE to select optimal token units.

Category: HowTo

What’s the difference between word-level and subword Vocabulary?

Word-level risks OOVs and large sizes; subword balances coverage and compactness using smaller units.

Category: Difference

How does Vocabulary impact embeddings?

Each token maps to an embedding; vocabulary size determines embedding matrix dimensions.

Category: Concept

What is a hot topic around Vocabulary?

Byte-level vocabularies simplify multilingual coverage and reduce tokenizer complexity.

Category: Topic

How do domain-specific Vocabularies help?

They improve tokenization and representation for specialized jargon, boosting downstream accuracy.

Category: Benefit

What are common mistakes when managing Vocabulary?

Using mismatched tokenizers and checkpoints, or overly small vocabularies causing fragmentation.

Category: Mistakes

Can Vocabulary be updated after training?

Yes, but it requires careful alignment with embeddings or adapter techniques; otherwise performance may degrade.

Category: Management