DataTerminology.comNEW
Also known as: MLM
A language model trained to predict masked tokens using bidirectional context in the input text.
DESCRIPTION
Masked language models replace a subset of tokens with a special mask and learn to predict the originals from context. This self-supervised objective enables bidirectional understanding, unlike causal models that predict next tokens. BERT popularized MLM pretraining for downstream tasks via fine-tuning. Variants use span masking and dynamic masking. MLMs excel in fill-in-the-blank tasks, representation learning, and feature extraction. They are foundational in modern NLP pipelines and transfer learning.
ALTERNATIVE DEFINITIONS
Wikipedia
In natural language processing, a masked language model (MLM) is a language model that receives an input where some percentage of tokens have been replaced by a mask token and attempts to predict those masked tokens.
Specifies the masking mechanism and prediction objective.
BERT Paper
We use a masked language model (MLM) objective, where we mask 15% of the tokens in each sequence at random, and then predict the masked tokens.
Original formulation with specific masking proportion.
Hugging Face
Masked language modeling consists of masking tokens in an input sequence and training a model to predict the masked tokens.
Task-oriented description for practical implementation.
RELATED TERMS
T5 Pretraining Tokenization Transformer Model Fine-Tuning Next-Token Prediction Span Masking Self-Supervised Learning
COMPETING TERMS
RELATIONS
- Broader: Language Model
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Masked language model
- Hugging Face - Tutorial on MLM task
- TensorFlow - BERT fine-tuning tutorial
- PyTorch - Transformer tutorial foundations
TOP QUESTIONS
What makes a Masked Language Model different from a Causal Language Model?
MLMs use bidirectional context to predict masked tokens, while causal models predict next tokens using left-to-right context.
Category: Misunderstanding
Why did BERT popularize Masked Language Models?
BERT demonstrated strong transfer learning via MLM pretraining, yielding state-of-the-art results on many NLP tasks through fine-tuning.
Category: Relevancy
How are tokens selected for masking in MLM?
Typically a random subset (e.g., 15%) of tokens is masked, sometimes with variations like span masking and dynamic masking strategies.
Category: Requirements
What tasks benefit from MLM pretraining?
Text classification, question answering, named entity recognition, and general feature extraction often benefit from MLM representations.
Category: Overview
What are risks when training an MLM?
Overfitting to pretraining domain, masking leakage artifacts, and high compute costs can reduce generalization and practical deployment value.
Category: Risk
How is MLM implemented with Transformers?
Introduce [MASK] tokens in inputs, train with cross-entropy over masked positions, and fine-tune the encoder for downstream tasks.
Category: Management
What is a current hot topic around MLMs?
Efficient pretraining with distillation and low-rank adapters, and combining MLM with instruction tuning for multitask performance.
Category: Topic
What competes with MLM for pretraining?
Causal language modeling powers generative LLMs; alternative objectives include denoising autoencoding and span corruption (e.g., T5).
Category: Alternatives
Where should I start to learn MLMs?
Read the BERT paper, follow Hugging Face tasks guide, and experiment with masking strategies on small corpora.
Category: Overview