🔁Encoder-Decoder

NEW

Also known as: Seq2Seq

Neural sequence-to-sequence architecture that encodes inputs to a representation and decodes it to generate outputs.

DESCRIPTION

Encoder–decoder architectures map variable-length inputs to outputs via an intermediate representation. Early seq2seq models used RNNs with attention; modern variants use transformer blocks with multi-head attention. They power tasks like translation, summarization, and text-to-text modeling (e.g., T5, BART). Training uses teacher forcing and decoding strategies like beam search for quality outputs.

ALTERNATIVE DEFINITIONS

Wikipedia

An encoder–decoder architecture is a neural network design pattern that uses an encoder to map input data to a fixed-dimensional vector and a decoder to map the vector to an output.

Defines the two-part structure and intermediate representation.

Source

arXiv (Bahdanau et al., 2015)

We propose a novel model that jointly learns to align and translate by introducing an attention mechanism that allows the model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word.

Introduces attention as a core enhancement to encoder–decoder translation.

Source

RELATED TERMS

Sequence-to-Sequence Transformer Model Attention Mechanism Machine Translation T5 BART Beam Search Teacher Forcing Encoder-only Models Decoder-only Models

COMPETING TERMS

Transformer Decoder-Only Encoder-only Transformer

RELATIONS

• Narrower: Autoencoder BART Transformer Encoder-Decoder

KNOWLEDGE AREAS

AI Data Science

TARGET AUDIENCE

AI Engineer Data Scientist MLOps Engineer

EXTERNAL SOURCES

• Wikipedia - Encoder–decoder architecture
• arXiv - Attention-based seq2seq paper
• arXiv - T5: text-to-text framework
• Hugging Face - Transformer encoder–decoder usage

TOP QUESTIONS

What are typical applications of an Encoder-Decoder architecture?

Machine translation, summarization, question answering, and style transfer are common. It excels at mapping sequences to sequences.

Category: Use cases

Why add attention to Encoder-Decoder models?

Attention lets decoders focus on relevant input positions, improving alignment and performance over fixed bottleneck vectors.

Category: Explanation

How does an Encoder-Decoder compare to decoder-only models?

Decoder-only models handle generation with causal masks, while encoder–decoders explicitly ingest source inputs, often excelling at supervised seq2seq tasks.

Category: Alternatives

What is beam search in Encoder-Decoder decoding?

Beam search explores multiple candidate sequences to improve output quality compared to greedy decoding.

Category: Definition

What pitfalls arise in training Encoder-Decoder models?

Exposure bias from teacher forcing and length bias can degrade quality. Scheduled sampling and length normalization help.

Category: Mistakes

How do I evaluate Encoder-Decoder outputs?

Use metrics like BLEU, ROUGE, or task-specific accuracy. Human evaluations remain critical for quality assessment.

Category: Testing

What is the role of positional encodings?

They inject order information into transformer blocks, enabling models to reason about token positions.

Category: Explanation

How does pretraining benefit Encoder-Decoder models?

Text-to-text pretraining (e.g., T5) transfers knowledge to diverse downstream tasks with minimal task-specific changes.

Category: Benefits

What hardware considerations matter for Encoder-Decoder inference?

Memory footprint for cross-attention can be significant. Use efficient attention and batching strategies.

Category: Requirements