DataTerminology.comNEW
An automatic evaluation metric for machine translation that measures n-gram precision against reference translations with a brevity penalty.
DESCRIPTION
BLEU (Bilingual Evaluation Understudy) compares machine translations against one or more reference translations using modified n-gram precision and a brevity penalty. It is corpus-level and language-independent, correlating reasonably with human judgments for some settings. BLEU is sensitive to tokenization and reference quality. Modern practice often uses sacreBLEU to standardize tokenization and reporting. Complementary metrics like METEOR, ROUGE, and learned metrics address BLEU’s limitations in semantics and recall.
ALTERNATIVE DEFINITIONS
Wikipedia
BLEU is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another.
High-level characterization of BLEU as an MT evaluation algorithm.
Papineni et al., 2002
We propose a method for automatic evaluation of machine translation that is quick to compute, inexpensive, language-independent, and correlates highly with human evaluation.
Introduces BLEU’s aims and correlation with human judgments.
RELATED TERMS
Machine Translation N-gram Precision Smoothing BLEURT SacreBLEU Tokenization Corpus BLEU Brevity Penalty
COMPETING TERMS
RELATIONS
KNOWLEDGE AREAS
AI Data Science Testing & Validation
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - BLEU
- Papineni et al. - Original BLEU paper
- sacreBLEU - Standardized BLEU implementation
- TensorFlow Text - BLEU metric usage guide
TOP QUESTIONS
What does BLEU Score measure in translation?
It measures n-gram precision against references, penalizing overly short outputs via a brevity penalty.
Category: Clarification
Why is sacreBLEU recommended for BLEU Score reporting?
It standardizes tokenization and parameters, improving reproducibility and comparability across papers and systems.
Category: BestPractice
Is BLEU Score reliable for sentence-level evaluation?
BLEU is designed for corpus-level evaluation; sentence-level BLEU is noisy without smoothing.
Category: Misunderstanding
What are BLEU Score limitations?
It focuses on surface n-grams, missing paraphrase and semantic adequacy; complement with learned or recall-oriented metrics.
Category: Risk
How many references improve BLEU Score evaluation?
Multiple references mitigate lexical mismatch, often improving correlation with human judgments.
Category: Opinion
How to improve BLEU Score fairly?
Optimize models and decoding but keep tokenization, case handling, and evaluation settings consistent and transparent.
Category: HowTo
What is the brevity penalty in BLEU Score?
It penalizes hypotheses shorter than references to discourage trivial high-precision short outputs.
Category: Clarification
How does decoding affect BLEU Score?
Beam search and length normalization can influence n-gram matches and length balance, impacting BLEU.
Category: Example
What metrics complement BLEU Score?
METEOR, ROUGE, chrF, and learned metrics like BLEURT or COMET capture recall and semantics.
Category: Alternatives