DataTerminology.comNEW
Also known as: Encoder–Decoder Transformer, Seq2Seq Transformer
A sequence-to-sequence transformer with an encoder that processes inputs and a decoder that generates outputs using self- and cross-attention.
DESCRIPTION
Encoder–decoder transformers separate input encoding from output generation, making them well-suited for tasks like translation, summarization, and text-to-text transfer. The encoder builds contextual representations; the decoder uses self-attention and cross-attention to condition on encoded inputs while generating tokens. Pretrained models like T5 and BART leverage denoising or text-to-text objectives. They often yield strong performance on supervised seq2seq tasks. Decoding commonly uses beam search with length penalties for fluent outputs.
ALTERNATIVE DEFINITIONS
Vaswani et al. (2017)
The model architecture follows the standard encoder–decoder structure.
Establishes the canonical architecture introduced by the Transformer paper.
Wikipedia
The original transformer uses an encoder–decoder architecture, with an encoder mapping an input sequence to a representation and a decoder generating an output sequence.
Summarizes functional roles of encoder and decoder in the architecture.
Hugging Face
Encoder-decoder models encode the input sequence into a hidden representation and decode it to generate the target sequence, often used for translation and summarization.
Highlights typical tasks and data flow in encoder–decoder models.
RELATED TERMS
Transformer Model Self-Attention Cross-Attention Machine Translation Sequence-to-Sequence BART T5 Positional Encoding Tokenization Beam Search
COMPETING TERMS
RELATIONS
- Broader: Encoder-Decoder
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer Software Architect Software Developer
EXTERNAL SOURCES
- Wikipedia - Transformer (machine learning)
- Vaswani et al. - Transformer introduction
- Hugging Face - T5 encoder–decoder docs
TOP QUESTIONS
When should I prefer an Encoder-Decoder Transformer?
Choose it for supervised sequence-to-sequence tasks like translation and summarization. It cleanly conditions outputs on encoded inputs for better structure.
Category: Application
How does cross-attention work in this architecture?
Cross-attention lets the decoder attend to encoder outputs, injecting input information into each decoding step.
Category: Concept
What are common training objectives?
Teacher-forced maximum likelihood, denoising autoencoding (BART), and text-to-text pretraining (T5) are common.
Category: Technique
What decoding methods are typical?
Beam search with length penalty is common for structured outputs; sampling works for creative tasks but can reduce consistency.
Category: Technique
How does it compare to Decoder-Only models?
Encoder–decoder excels at seq2seq tasks; decoder-only shines at open-ended generation and few-shot prompting.
Category: Comparison
What are pitfalls in deployment?
Latency can be higher due to cross-attention. Optimize with caching, batching, and pruning if needed.
Category: Mistakes
What is the role of positional encoding here?
It provides order information to both encoder and decoder since attention is permutation-invariant.
Category: Concept
How do pretraining strategies affect performance?
Denoising and text-to-text pretraining improve transfer to downstream tasks, especially with multitask fine-tuning.
Category: Opinion
Where should I start learning this architecture?
Read the original Transformer paper, then explore T5 and BART documentation and tutorials.
Category: Getting Started