DataTerminology.comNEW
Also known as: Encoder-decoder attention
An attention mechanism that lets one sequence attend to another sequence’s representations to guide decoding.
DESCRIPTION
Cross-attention, also called encoder–decoder attention, computes attention from decoder queries to encoder keys/values. It enables the decoder to focus on relevant source positions, improving translation, summarization, and multimodal fusion. Together with self-attention and feed-forward layers, it forms the core of Transformer decoders. Cross-attention generalizes to multimodal models by attending across modalities.
ALTERNATIVE DEFINITIONS
Wikipedia
The decoder uses an encoder–decoder attention mechanism (cross-attention) which allows the decoder to attend to the encoder outputs.
Defines cross-attention in the context of the Transformer decoder.
Vaswani et al.
Encoder–decoder attention layers allow the decoder to attend to the encoder output, enabling the model to focus on appropriate parts of the input sequence.
Original paper description of encoder–decoder attention’s role.
RELATED TERMS
Transformer Model Attention Head Query Key Values Encoder Decoder Attention Mask Encoder-Decoder
COMPETING TERMS
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer Software Developer
EXTERNAL SOURCES
- Wikipedia - Transformer (machine learning)
- Vaswani et al. - Transformer original paper
- Jay Alammar - Visual explanation of attention
TOP QUESTIONS
Why is Cross-Attention crucial in sequence-to-sequence models?
It lets the decoder select relevant encoder states, improving alignment and translation quality over fixed context vectors.
Category: Relevancy
How does Cross-Attention differ from Self-Attention?
Self-attention attends within the same sequence; cross-attention attends from the decoder to the encoder outputs.
Category: Comparison
Can Cross-Attention be used for multimodal models?
Yes. Text decoders can attend to image embeddings, enabling captioning and visual question answering.
Category: Application
What are common implementation pitfalls for Cross-Attention?
Incorrect masking or dimension mismatches can break alignment. Careful tensor shapes and masking are essential.
Category: Mistakes
How many heads should Cross-Attention use?
It depends on model width and task. Multiple heads capture diverse alignment patterns; tune empirically.
Category: BestPractices
Does Cross-Attention increase inference cost?
Yes, it adds attention over encoder states. Efficient caching and pruning can mitigate overhead.
Category: Performance
Is Cross-Attention relevant in decoder-only LLMs?
Not in standard form, but analogous cross-modal attention arises when conditioning on external encoders (e.g., vision encoders).
Category: Scope
How does Cross-Attention improve summarization?
It prioritizes salient parts of the source document, yielding focused and faithful summaries.
Category: Example
What’s the biggest misunderstanding about Cross-Attention?
That it replaces self-attention; in practice, both are complementary in Transformer decoders.
Category: Misunderstanding