DataTerminology.comNEW
Also known as: Decoder-Only Transformer, Autoregressive Transformer
A transformer architecture composed only of decoder blocks with causal attention for autoregressive next-token prediction in language modeling.
DESCRIPTION
Decoder-only transformers model sequences by predicting the next token conditioned on previous tokens, enforced by causal (masked) self-attention. They underpin modern general-purpose LLMs like GPT variants used for generation, reasoning, and tool-use. Training typically involves large-scale language modeling with instruction tuning and reinforcement learning for alignment. They excel in open-ended generation but can be less efficient for sequence-to-sequence tasks compared to encoder–decoder models. Inference uses greedy, beam, or sampling-based decoding with temperature and top-k/p controls.
ALTERNATIVE DEFINITIONS
Wikipedia
In a decoder-only transformer, the model is trained to predict the next token in a sequence using masked self-attention that prevents attending to future tokens.
Highlights causal masking and next-token prediction as defining traits.
Hugging Face Course
Decoder-only models are trained to predict the next token given the previous tokens, using causal self-attention to avoid attending to future positions.
Educational framing focused on training objective and causal attention.
RELATED TERMS
Transformer Model Causal Attention Language Modeling GPT Self-Attention Tokenization Next-Token Prediction Pretraining Instruction Tuning RLHF
COMPETING TERMS
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer Software Architect Software Developer
EXTERNAL SOURCES
- Wikipedia - Transformer (machine learning)
- Hugging Face - Course on transformer models
- Vaswani et al. - Original Transformer paper
TOP QUESTIONS
Why choose a Decoder-Only Transformer for text generation?
Its autoregressive design suits open-ended generation and reasoning. It scales well with data and compute, achieving strong performance on diverse tasks.
Category: Relevancy
What’s the core training objective?
Next-token prediction under causal masking. This objective aligns closely with generative use cases.
Category: Concept
How does it differ from Encoder-Decoder?
Encoder-decoder models encode inputs explicitly, aiding seq2seq tasks like translation. Decoder-only models rely on context in the prompt itself.
Category: Comparison
What are common decoding strategies?
Greedy, beam search, top-k, and nucleus (top-p) sampling. Temperature adjusts randomness and diversity.
Category: Technique
What are typical pitfalls?
Prompt leakage, hallucinations, and context window limits. Guard with retrieval augmentation, guardrails, and monitoring.
Category: Mistakes
How do scaling laws impact Decoder-Only models?
Performance improves predictably with model, data, and compute scaling, subject to diminishing returns and optimization constraints.
Category: Opinion
Is it efficient for translation?
Encoder-decoder often outperforms for translation. Decoder-only can still work with properly formatted prompts and examples.
Category: Comparison
How is safety handled?
Through alignment techniques, post-training filters, and policy prompts. Continuous red teaming is recommended.
Category: Risks
Where to start learning Decoder-Only Transformers?
Read the original Transformer paper and Hugging Face course modules, then experiment with small GPT-like models.
Category: Getting Started