🧠Multi-Head Attention

NEW

Also known as: MHA

An attention mechanism that computes multiple parallel attention heads to capture diverse relationships across representation subspaces.

DESCRIPTION

Multi-head attention splits queries, keys, and values across multiple heads, each attending to different aspects of the input. Heads are concatenated and linearly projected to combine information. This mechanism improves representation power and enables parallel computation, central to transformer architectures in NLP and vision. It enhances long-range dependencies and contextual understanding. Typical configurations use 4–16 heads, balancing capacity and compute.

ALTERNATIVE DEFINITIONS

Vaswani et al.

Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.

Original paper highlights attending to multiple subspaces simultaneously.

Source

Jay Alammar

Multi-head attention is simply running attention multiple times in parallel.

Stresses intuitive parallelism of attention computations.

Source

RELATED TERMS

Attention Mechanism Transformer Model Scaled Dot-Product Attention Self-Attention Query-Key-Value Positional Encoding Encoder-Decoder Context Window Residual Connection Layer Normalization

COMPETING TERMS

Single-Head Attention Recurrent Neural Network

KNOWLEDGE AREAS

AI Data Science

TARGET AUDIENCE

AI Engineer Data Scientist MLOps Engineer

EXTERNAL SOURCES

• Wikipedia - Attention (machine learning)
• Hugging Face - Transformers library documentation
• TensorFlow - Keras MultiHeadAttention layer API
• PyTorch - PyTorch MultiheadAttention API

TOP QUESTIONS

What is Multi-head Attention?

It is an attention mechanism that computes multiple attention heads in parallel, capturing diverse relationships within data.

Category: Definition

Why use multiple heads in attention?

Multiple heads learn complementary patterns across representation subspaces, improving contextual understanding and model expressiveness.

Category: Relevancy

How is Multi-head Attention implemented?

Queries, keys, values are linearly projected into multiple heads, attention computed per head, then concatenated and projected back.

Category: Requirements

What are common mistakes with Multi-head Attention?

Using too many heads without sufficient width can underperform; improper initialization and scaling can destabilize training.

Category: Mistakes

What are the risks of Multi-head Attention?

Increased compute and memory overhead; potential overfitting if capacity is excessive relative to data.

Category: Risks

How does Multi-head compare to Single-head Attention?

Single-head may miss diverse relationships; multi-head captures multiple perspectives simultaneously.

Category: Alternatives

Where is Multi-head Attention used today?

It is core to transformers used in LLMs, machine translation, vision transformers, and speech models.

Category: Example

How many heads should be used?

Common choices are 4–16 heads; tune based on model size, data, and compute budget.

Category: Opinion

Where to start learning Multi-head Attention?

Read ‘Attention Is All You Need’ and explore Hugging Face and TensorFlow API docs.

Category: Overview