DataTerminology.comNEW
Also known as: MLM
A pretraining objective where randomly masked tokens are predicted from bidirectional context.
DESCRIPTION
Masked Language Modeling trains encoders by randomly masking tokens and predicting them from surrounding context. Introduced with BERT, it enables bidirectional representations that excel in understanding tasks. Variants include span masking (T5) and whole-word masking. MLM differs from causal objectives by leveraging both left and right context. It is foundational for tasks like classification, retrieval, and QA after fine-tuning.
ALTERNATIVE DEFINITIONS
BERT Paper (Devlin et al.)
We use a masked language model (MLM) pre-training objective, which randomly masks some of the input tokens and then predicts the masked tokens.
Canonical definition from the BERT paper.
Wikipedia
Masked language modeling is a fill-in-the-blank task where models are trained to predict deliberately masked tokens in a sequence.
Explains the task nature in accessible terms.
RELATED TERMS
BERT Denoising Objective Span Corruption Encoder Models Autoencoding Token Masking Self-Supervised Learning Perplexity Cloze Task Bidirectional Context
COMPETING TERMS
Next-Token Prediction
RELATIONS
- Broader: Language Modeling
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Masked language model
- arXiv - Original BERT paper with MLM
- Hugging Face - MLM task tutorial
TOP QUESTIONS
Why use Masked Language Modeling?
It yields strong bidirectional representations for understanding tasks like classification and QA.
Category: Benefits
How much masking should be applied in MLM?
BERT commonly masks 15% of tokens; tune rates based on dataset and objective variants.
Category: BestPractice
What’s the difference between MLM and Next Token Prediction?
MLM uses bidirectional context and predicts masked tokens; next-token prediction uses left context only.
Category: Alternatives
What is span masking in MLM variants?
Contiguous spans are masked and predicted, aligning with T5’s denoising objectives.
Category: Definition
Does MLM directly support text generation?
No, MLM encoders are not generative; generation typically requires seq2seq or causal decoders.
Category: Misunderstanding
Which models use MLM?
BERT, RoBERTa, DeBERTa, and ELECTRA variants (with related pretext tasks).
Category: Example
What are common pitfalls in MLM training?
Over-masking, small context windows, and insufficient corpus diversity can limit representation quality.
Category: Mistakes
How to evaluate MLM-pretrained models?
Fine-tune on downstream tasks and report standard benchmarks like GLUE, SQuAD, or custom metrics.
Category: Testing
Can MLM help multilingual understanding?
Yes, multilingual MLM (e.g., mBERT, XLM-R) learns cross-lingual representations from shared subword vocabularies.
Category: Benefits