DataTerminology.comNEW
Also known as: Extended Context, Long-Context Window, Large Context Window
Support for very long input sequences, enabling models to utilize distant information during inference.
DESCRIPTION
Long context refers to models capable of processing extended input sequences, often tens or hundreds of thousands of tokens. It improves tasks involving documents, codebases, or multi-turn conversations by retaining distant details. Achieving long context requires efficient attention mechanisms, memory strategies, and careful prompt design. Trade-offs include latency, cost, and potential dilution of relevant information. Alternatives like RAG provide targeted retrieval without expanding the context window.
ALTERNATIVE DEFINITIONS
Anthropic
We’re expanding Claude’s context window to 100,000 tokens, allowing it to process long documents, code, and conversations in a single prompt.
Defines a concrete context size and use cases like long documents and code.
OpenAI
Large context models can accept inputs with significantly more tokens, enabling better comprehension of extended text.
Emphasizes token capacity and comprehension over extended text spans.
RELATED TERMS
Transformer (Machine Learning) Context Window Retrieval-Augmented Generation Chunking Summarization Attention Mechanism Memory Augmentation Windowed Attention Sliding Window
COMPETING TERMS
Retrieval-Augmented Generation (RAG)
RELATIONS
- Broader: Context
KNOWLEDGE AREAS
AI Infrastructure Platform & Tooling Data Engineering
TARGET AUDIENCE
AI Engineer Data Scientist Software Developer MLOps Engineer Data Architect
EXTERNAL SOURCES
- Wikipedia - Transformer (machine learning model)
- Anthropic - Announcement of 100k context
- OpenAI - Guide to large context usage
- arXiv - Longformer: efficient long sequences
TOP QUESTIONS
What is long context in language models?
It’s the ability to process very long input sequences, retaining and using distant information during inference.
Category: Definition
Why is long context useful?
It enables understanding of long documents, codebases, and multi-turn conversations without external retrieval.
Category: Relevancy
What are the trade-offs of long context?
Higher latency, cost, and potential dilution of relevant details. Memory and attention efficiency are critical.
Category: Risks
How does long context compare to RAG?
Long context embeds information directly in prompts; RAG fetches relevant snippets on demand. RAG can be cheaper and more precise.
Category: Alternatives
What architectures support long context?
Efficient attention variants like Longformer, BigBird, and windowed/sliding attention enable longer sequence handling.
Category: Requirements
How to prompt effectively with long context?
Use structure, headings, and chunking. Place critical information near the end or beginning depending on model recency bias.
Category: Best practices
What evaluation methods suit long context?
Benchmarks with long documents or code, recall of distant facts, and latency and cost profiling.
Category: Testing
Can long context models forget relevant information?
Yes. Attention dispersion and recency bias may reduce recall. Summarization and anchors help.
Category: Misunderstanding
What are typical context sizes today?
Modern models range from 16k to 200k+ tokens, depending on architecture and vendor.
Category: Topic