DataTerminology.comNEW
Also known as: ALiBi
A positional bias technique for attention that enables models to extrapolate to longer sequences.
DESCRIPTION
Attention with Linear Biases (ALiBi) adds a distance-dependent linear bias to attention scores, reducing reliance on learned absolute position embeddings. This simple modification allows transformers trained on shorter sequences to generalize to longer contexts at inference time. ALiBi avoids resizing position embeddings and maintains computational efficiency. It has been used in large language models to extend context windows without full retraining. Compared to RoPE and relative encodings, ALiBi emphasizes monotonic distance penalties directly in attention logits.
ALTERNATIVE DEFINITIONS
arXiv
We show that Transformers can extrapolate to sequence lengths longer than those seen during training by adding attention biases that are a linear function of distance between tokens.
Highlights extrapolation benefit via simple linear distance-based bias.
GitHub (Ofir Press)
ALiBi lets pretrained models process sequences longer than those used during training by adding a simple linear bias to attention scores.
Practical, implementation-oriented phrasing by the authors.
RELATED TERMS
Transformer Model Positional Encoding Self-Attention Long-Context Modeling Causal Attention Extrapolation Language Model Sequence Modeling
COMPETING TERMS
Rotary Positional Embeddings Sinusoidal Positional Encoding Relative Positional Encoding
RELATIONS
- Broader: Positional Encoding
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer Software Developer
EXTERNAL SOURCES
- Wikipedia - Transformer (machine learning model)
- arXiv - Original ALiBi research paper
- Ofir Press - Reference implementation and details
TOP QUESTIONS
Why is Attention with Linear Biases (ALiBi) useful for long-context transformers?
ALiBi enables extrapolation to longer sequences without retraining positional embeddings. This reduces retraining cost and simplifies extending context windows.
Category: Relevancy
How does ALiBi differ from Rotary Positional Embeddings?
ALiBi penalizes attention by distance via linear biases in logits, while RoPE rotates queries/keys in a complex plane to encode relative positions.
Category: Comparison
Does ALiBi increase compute cost?
No, it minimally changes attention score computation by adding simple biases, preserving the asymptotic complexity of standard attention.
Category: Performance
Can ALiBi be added to existing pretrained models?
Yes, many implementations retrofit ALiBi by modifying attention calculations, enabling longer inputs at inference.
Category: Adoption
What tasks benefit most from ALiBi?
Tasks requiring long-range dependencies such as long-document QA, code completion with long files, or retrieval-augmented generation benefit.
Category: UseCase
What are the trade-offs of ALiBi?
While simple and effective for extrapolation, ALiBi may underperform on tasks needing fine-grained absolute positions compared to some relative schemes.
Category: Tradeoff
Is ALiBi compatible with causal attention?
Yes, ALiBi is commonly used with causal masks, adding distance biases only within the allowed attention window.
Category: Compatibility
How does ALiBi impact training dynamics?
It reduces dependency on learned positional embeddings, often stabilizing generalization to unseen lengths without changing core optimization.
Category: Theory
What are alternatives to ALiBi for long context?
Rotary Positional Embeddings, relative positional encodings, position interpolation, and sparse/efficient attention mechanisms are alternatives.
Category: Alternatives