DataTerminology.comNEW
Also known as: Bilingual Evaluation Understudy
An automatic metric that scores machine translation quality by n-gram overlap with reference texts, adjusted by a brevity penalty.
DESCRIPTION
BLEU evaluates how closely a candidate translation matches reference translations using modified n-gram precision. A brevity penalty discourages overly short outputs. Scores range from 0 to 1 (often reported as 0–100). It is fast and language-agnostic, enabling reproducible benchmarking. Limitations include insensitivity to synonyms and semantics; newer metrics like COMET address such issues. SacreBLEU standardizes tokenization and computation to improve comparability.
ALTERNATIVE DEFINITIONS
ACL Anthology (Papineni et al., 2002)
We present a method for automatic evaluation of machine translation that correlates highly with human judgments of quality. Our method, BLEU, is quick to calculate, inexpensive, and language-independent.
Original paper emphasizes correlation with human judgments and efficiency.
Wikipedia
BLEU is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another.
Concise general-purpose definition for MT evaluation.
Hugging Face
SacreBLEU is a standard BLEU implementation that avoids common pitfalls and ensures comparable results across experiments.
Focuses on standardized computation to improve reproducibility.
RELATED TERMS
Machine Translation N-gram ROUGE Score SacreBLEU Precision Brevity Penalty
COMPETING TERMS
RELATIONS
- Broader: BLEU Score
KNOWLEDGE AREAS
AI Data Science Testing & Validation MLOps
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - BLEU
- SacreBLEU - Reference BLEU implementation
- WMT - MT metrics shared task
- ACL Anthology - Original BLEU paper
TOP QUESTIONS
What does the BLEU brevity penalty do?
It penalizes overly short translations that achieve high precision by omitting content, encouraging adequate length.
Category: Understanding
Why is BLEU widely used for machine translation?
It’s fast, language-agnostic, and correlates reasonably with human judgments, enabling large-scale benchmarking.
Category: Relevancy
What are the main limitations of BLEU?
It is precision-focused, insensitive to synonyms and meaning, and can be gamed by tokenization choices without standardization.
Category: Risks
How can I ensure reproducible BLEU scores?
Use SacreBLEU with fixed tokenization settings and document evaluation datasets and parameters.
Category: BestPractice
How does BLEU compare to METEOR or COMET?
METEOR incorporates synonyms and recall; COMET uses learned models and often correlates better with human judgments.
Category: Alternatives
What n-gram orders are typical in BLEU?
BLEU commonly uses up to 4-grams with geometric averaging of modified precisions.
Category: Understanding
Is BLEU suitable for summarization evaluation?
It can be used, but ROUGE or learned metrics often align better with summarization quality.
Category: Opinion
How many reference translations should I use with BLEU?
Multiple references can improve robustness, but many benchmarks rely on one due to resource limits.
Category: Methods
What’s a hot topic around BLEU today?
Shifting to learned metrics (e.g., COMET) and task-specific evaluations for better human correlation.
Category: Topic