DataTerminology.comNEW
Attention heads that copy-and-continue token sequences, implementing an induction algorithm that extends patterns seen earlier in the context.
DESCRIPTION
Induction heads are a mechanistic pattern in transformers where specific attention heads learn to continue repeated sequences. They attend from a current token to a previous matching token and then to the subsequent token, effectively copying patterns forward. This behavior helps explain few-shot generalization, code completion, and pattern following. Identifying induction heads informs interpretability, pruning decisions, and prompt design, though not all generalization relies on them.
ALTERNATIVE DEFINITIONS
Anthropic
We observe induction headsāattention heads that implement an induction algorithm by copying sequences forward to continue patterns.
Defines induction heads by their copy-and-continue algorithmic behavior.
arXiv
Induction heads are specialized attention heads that attend to previous occurrences of a token sequence to predict its continuation.
Stresses attention to previous occurrences enabling continuation.
RELATED TERMS
Self-Attention Transformer Model In-Context Learning Mechanistic Interpretability Attention Head Copy Mechanism Positional Encoding Next-Token Prediction Transformer Circuits Pattern Completion
COMPETING TERMS
N-gram Language Model
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer
EXTERNAL SOURCES
- Anthropic - Original analysis introducing induction heads
- Neel Nanda - Educational resources on transformer circuits
- arXiv - Papers studying in-context mechanisms
TOP QUESTIONS
How do Induction Heads work in transformers?
They attend from the current token to a previous matching token and then leverage that to predict the next token, continuing the sequence.
Category: Example
Why are Induction Heads important?
They explain how transformers generalize patterns in-context, illuminating few-shot learning and code completion behaviors.
Category: Relevancy
How can I detect Induction Heads?
Use attention pattern analysis, activation patching, and ablation to find heads that implement copy-and-continue behavior.
Category: HowTo
Are Induction Heads the only mechanism behind in-context learning?
No, they are one mechanism; other heads and layers contribute to reasoning, memory, and structure following.
Category: Misunderstanding
What risks exist when modifying Induction Heads?
Pruning or altering them can degrade generalization, destabilize training, or harm domain tasks like coding.
Category: Risks
How do Induction Heads relate to n-gram models?
Both capture local repetition patterns, but induction heads emerge from learned attention, not fixed n-gram statistics.
Category: Alternatives
Do all transformer models learn Induction Heads?
They commonly emerge in trained LMs, but prevalence and strength vary by data, scale, and training regime.
Category: Opinion
Can prompt design leverage Induction Heads?
Yes, repeating relevant patterns or providing in-context exemplars can encourage pattern continuation.
Category: HowTo
What tools help analyze Induction Heads?
Mechanistic interpretability toolkits, e.g., transformer lens, provide attention visualization and intervention methods.
Category: Requirements