DataTerminology.comNEW
Also known as: Text Chunking, Document Chunking
Splitting large documents into smaller, semantically coherent segments to improve indexing, retrieval, and model context fit.
DESCRIPTION
Chunking creates manageable segments from long texts or datasets for embedding, retrieval, and LLM context windows. Good chunks balance size with semantic cohesion, often using overlap and natural boundaries like headings or sentences. Strategies vary by task: RAG prefers semantically dense chunks; summarization may use hierarchical chunking. Overly large or arbitrary chunks reduce retrieval precision, while too-small chunks lose context. Tools provide tunable splitters based on tokens, sentences, or recursive rules.
ALTERNATIVE DEFINITIONS
Weaviate
Chunking is the process of splitting documents into smaller pieces (chunks) to improve retrieval performance and fit model context limits.
Focuses on retrieval performance and context window constraints.
LangChain
Text splitters create chunks of text that are small enough to fit within a model’s context window while preserving as much meaning as possible.
Emphasizes context window limits and semantic preservation.
RELATED TERMS
Text Splitting Retrieval-Augmented Generation Vector Embeddings Sliding Window Overlap Passage Retrieval Indexing Context Window Document Segmentation
COMPETING TERMS
Tokenization Sentence Segmentation
KNOWLEDGE AREAS
AI Data Science Data Engineering MLOps
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer Data Engineer Analytics Engineer
EXTERNAL SOURCES
- Wikipedia - Text segmentation
- Pinecone - Chunking strategies for RAG
- OpenAI - Retrieval guidance and chunking
- Milvus - Embedding and chunking pipeline
TOP QUESTIONS
Why is Chunking important for retrieval-augmented generation?
It creates semantically coherent units that improve embedding quality and retrieval precision while respecting model context limits.
Category: Relevancy
What chunk size should I use for Chunking?
Common sizes range from 200–800 tokens with 10–20% overlap. Tune based on task, model, and domain.
Category: BestPractices
How does Chunking differ from Tokenization?
Tokenization breaks text into tokens for models; chunking groups content into larger segments for retrieval and context handling.
Category: Misunderstanding
What are common mistakes in Chunking?
Using arbitrary fixed-length splits that ignore semantics, or insufficient overlap causing context loss across chunks.
Category: Mistakes
What’s a good starting strategy for Chunking?
Use recursive splitters by headings and sentences, then enforce token limits with modest overlap.
Category: Requirements
How do I evaluate Chunking choices?
Measure retrieval precision/recall and downstream task accuracy; iterate with A/B tests on chunk size and overlap.
Category: Testing
Does Chunking help with multilingual data?
Yes. Respect language boundaries and sentence segmentation for better semantic coherence.
Category: BestPractices
How does Chunking affect cost and latency?
Smaller chunks increase index size and retrieval fan-out; larger chunks increase prompt tokens. Balance for throughput and cost.
Category: Risks
What tools support Chunking out of the box?
LangChain, LlamaIndex, Haystack, and vector DBs provide configurable splitters.
Category: Tools