🔁Encoder-Decoder Architecture

NEW

Also known as: Seq2Seq Architecture, RNN Encoder–Decoder

A neural modeling pattern where an encoder compresses inputs to a representation that a decoder expands into the target sequence.

DESCRIPTION

Encoder-decoder models map variable-length inputs to outputs via an intermediate representation. Early seq2seq used RNNs or LSTMs; attention mechanisms improved long-range dependencies. Transformers generalize the pattern with attention-based encoders and decoders. Applications include machine translation, summarization, speech recognition, and image captioning. Training uses teacher forcing and sequence-level objectives; decoding leverages beam search or sampling. Variants include pointer-generator networks, copy mechanisms, and non-autoregressive decoders.

ALTERNATIVE DEFINITIONS

Cho et al., 2014

We propose a novel neural network model (RNN Encoder–Decoder) that consists of two recurrent neural networks that work together to encode a sequence into a fixed-length vector representation and to decode the representation into another sequence.

Defines the core encoder-decoder concept in sequence learning.

Source

Wikipedia

Sequence-to-sequence models are a class of models that transform sequences to sequences, typically using an encoder–decoder architecture.

Links seq2seq and encoder–decoder as the standard formulation.

Source

RELATED TERMS

Sequence-to-Sequence Attention Mechanism Transformer Model Recurrent Neural Network LSTM Machine Translation Bahdanau Attention Teacher Forcing Beam Search Autoencoder

COMPETING TERMS

Decoder-only Architecture Connectionist Temporal Classification

KNOWLEDGE AREAS

AI Data Science

TARGET AUDIENCE

AI Engineer Data Scientist MLOps Engineer Software Developer

EXTERNAL SOURCES

• Wikipedia - Sequence-to-sequence
• Bahdanau et al. - Attention in encoder-decoder NMT
• Cho et al. - RNN encoder–decoder paper
• Goodfellow, Bengio, Courville - Textbook coverage of seq2seq

TOP QUESTIONS

Why use an Encoder-Decoder Architecture?

It elegantly handles variable-length sequence mapping with learned intermediate representations and flexible attention mechanisms.

Category: Relevancy

How does attention improve Encoder-Decoder models?

Attention lets the decoder focus on relevant encoder states dynamically, improving long-range dependencies and accuracy.

Category: Clarification

What tasks fit the Encoder-Decoder Architecture?

Translation, summarization, dialogue, speech recognition, and image captioning are common applications.

Category: Scope

What is teacher forcing in Encoder-Decoder training?

It feeds ground-truth tokens to the decoder during training to stabilize learning and speed convergence.

Category: Clarification

How does Encoder-Decoder compare to Decoder-only models?

Decoder-only models condition on context to generate outputs, while encoder-decoder explicitly encodes inputs for conditional generation.

Category: Comparison

What decoding methods are used with Encoder-Decoder models?

Beam search, top-k sampling, and nucleus sampling trade off quality and diversity.

Category: HowTo

What are common pitfalls with Encoder-Decoder training?

Exposure bias from teacher forcing and length bias in decoding often require scheduled sampling or length normalization.

Category: Mistakes

How do copy mechanisms extend Encoder-Decoder models?

They allow direct copying of source tokens, improving handling of rare or out-of-vocabulary items.

Category: Topic

What sparked the Encoder-Decoder approach?

Early seq2seq work showed RNNs can encode inputs to fixed vectors and decode outputs, later augmented by attention and Transformers.

Category: Origin