📏Positional Encoding

NEW

Also known as: Positional embedding

A method to inject token position information into transformer inputs using fixed or learned vectors.

DESCRIPTION

Positional encoding provides order awareness to attention-based models that lack recurrence or convolution. Sinusoidal encodings add fixed, frequency-based signals, while learned embeddings parameterize positions directly. Relative schemes (e.g., RoPE, ALiBi) improve extrapolation and long-context performance by modeling pairwise distances or rotations in attention. Choosing the scheme affects generalization to longer sequences, memory, and model efficiency.

ALTERNATIVE DEFINITIONS

Vaswani et al. (Attention Is All You Need)

We add sine and cosine functions of different frequencies to the input embeddings.

Defines the original sinusoidal positional encoding used in transformers.

Source

Su et al. (RoFormer: Enhanced Transformer with Rotary Position Embedding)

We propose Rotary Position Embedding (RoPE), which encodes absolute position with a rotation matrix that naturally incorporates relative position information.

Introduces a rotation-based scheme that supports relative position modeling.

Source

Press et al. (Train Short, Test Long: Attention with Linear Biases)

We add linear biases to attention scores dependent on token distance, enabling models trained on short sequences to generalize to longer ones.

Presents ALiBi linear biases for long-context generalization without learned position embeddings.

Source

RELATED TERMS

Transformer Model Self-Attention Sinusoidal Positional Encoding Learned Positional Embeddings Attention with Linear Biases Token Embedding Sequence Modeling Attention Mechanism

COMPETING TERMS

Relative Positional Encoding Rotary Positional Embeddings

RELATIONS

• Broader: Encoding
• Narrower: Attention with Linear Biases Sinusoidal Positional Encoding

KNOWLEDGE AREAS

AI Data Science Mathematics

TARGET AUDIENCE

AI Engineer Data Scientist MLOps Engineer

EXTERNAL SOURCES

• Wikipedia - Transformer (machine learning model)
• Vaswani et al. - Original transformer positional encoding
• Su et al. - RoPE positional embedding method
• Press et al. - ALiBi linear position bias
• Hugging Face - BERT uses absolute position embeddings

TOP QUESTIONS

What is Positional Encoding in transformers?

It is a mechanism to supply token order information using fixed or learned vectors, enabling attention to reason about sequence positions.

Category: Definition

Why do we need Positional Encoding?

Self-attention is permutation-invariant; positional encoding provides order awareness critical for language and sequence tasks.

Category: Relevancy

How does sinusoidal Positional Encoding work?

It adds sine and cosine signals at multiple frequencies to embeddings so positions are encoded continuously and extrapolate to longer sequences.

Category: HowTo

What is the difference between absolute and relative Positional Encoding?

Absolute encodes positions directly, while relative models pairwise distances, often improving long-context generalization.

Category: Difference

What are RoPE and ALiBi in Positional Encoding?

RoPE encodes position via rotations in attention; ALiBi adds linear biases to attention scores based on token distance.

Category: Example

What are common mistakes with Positional Encoding?

Using learned embeddings with limited training lengths can hurt extrapolation; mis-scaling encodings may destabilize training.

Category: Mistakes

What risks exist when choosing a Positional Encoding scheme?

Poor choices may impair long-range understanding, degrade generalization, or increase compute without clear benefits.

Category: Risks

How does Positional Encoding impact attention heads?

It guides heads to capture distance and order-dependent patterns, improving sequence modeling accuracy.

Category: Relation

Where should I start learning about Positional Encoding?

Read the transformer paper’s section on positional encoding, then explore RoPE and ALiBi for modern alternatives.

Category: Resources