DataTerminology.comNEW
Also known as: Intra-attention
Mechanism where each token weights and aggregates all tokens in the same sequence to build context-aware representations.
DESCRIPTION
Self-attention computes pairwise interactions within a sequence so each element can contextualize itself using all others. It underpins the Transformer architecture by replacing recurrence with parallelizable matrix operations. Queries, keys, and values are learned projections that produce attention weights, often normalized by softmax. Multi-head attention captures different relational aspects in parallel. Masked self-attention enables autoregressive generation by preventing peeking at future tokens. While powerful, self-attention has quadratic complexity in sequence length, prompting research into sparse, linear, and memory-efficient variants. It is widely used across NLP, vision, audio, and multimodal models. Proper positional encodings are essential since attention is permutation-invariant by default.
ALTERNATIVE DEFINITIONS
Vaswani et al., Attention Is All You Need
An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors.
Provides the formal QKV mapping foundational to attention and self-attention.
Wikipedia
Self-attention, also called intra-attention, is an attention mechanism relating different positions of a single sequence to compute a representation of the sequence.
Emphasizes intra-sequence relationships and representation learning.
Stanford CS224n
Self-attention computes attention over a single sequence by relating different positions of that sequence to each other.
Pedagogical framing focused on NLP and transformer encoders.
The Illustrated Transformer
Self-attention allows the model to look at other positions in the input sequence as it encodes a given word.
Intuitive visualization-oriented explanation of self-attention behavior.
RELATED TERMS
Transformer Model Attention Mechanism Scaled Dot-Product Attention Attention Head Query-Key-Value Positional Encoding Masked Attention Cross-Attention Residual Connection Layer Normalization
COMPETING TERMS
Recurrent Neural Network Convolutional Neural Network
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer Software Developer
EXTERNAL SOURCES
- Wikipedia - Attention (machine learning)
- Vaswani et al. - Original transformer paper defining attention.
- PyTorch - API for multi-head attention layer.
- TensorFlow - Transformer tutorial with attention code.
- Stanford CS224n - Lecture notes on attention and transformers.
TOP QUESTIONS
Why is Self-attention central to transformer performance?
Self-attention models all token interactions in parallel, enabling rich context capture and efficient GPU utilization. This yields superior scalability and long-range dependency modeling versus recurrent networks.
Category: Relevancy
How does Self-attention differ from Cross-attention?
Self-attention attends within a single sequence, while cross-attention attends between two sequences (e.g., decoder queries encoder outputs). They often co-exist in encoder–decoder architectures.
Category: Comparison
What is the computational cost of Self-attention?
Vanilla self-attention scales quadratically with sequence length due to pairwise interactions. Techniques like sparse attention, linear attention, and FlashAttention reduce memory and compute overhead.
Category: Performance
How is masking used in Self-attention?
Causal masks prevent attention to future positions in autoregressive generation; padding masks ignore padded tokens. Incorrect masking often causes leakage or degraded training.
Category: Practice
Where is Self-attention applied beyond NLP?
It powers vision transformers, audio models, recommendation, and multimodal systems. Self-attention’s flexibility makes it a general-purpose building block across domains.
Category: Applications
What is Multi-Head Self-attention?
Multiple attention heads learn different relational subspaces in parallel, improving expressiveness and stability. Outputs are concatenated and projected to mix information.
Category: Concept
What are common mistakes when implementing Self-attention?
Mismatched dimensions for Q, K, V, missing scaling factor, and improper masking are frequent. Numerical instability can arise without proper softmax scaling and normalization.
Category: Mistakes
What are alternatives to Self-attention for long sequences?
Recurrent models, convolutions, state-space models, and structured/sparse attention are options. Choice depends on latency, memory, and domain constraints.
Category: Alternatives
Who introduced Self-attention in its modern form?
Vaswani and colleagues formalized scaled dot-product self-attention in the 2017 “Attention Is All You Need” paper, catalyzing the Transformer era.
Category: Origin