📏METEOR

NEW

Also known as: Metric for Evaluation of Translation with Explicit ORdering

A machine translation evaluation metric using alignment-based unigram matches with stemming, synonyms, and penalties to better correlate with human judgments.

DESCRIPTION

METEOR evaluates MT output by aligning hypothesis and reference words using exact, stem, synonym, and paraphrase matches. It computes a harmonic mean of precision and recall (favoring recall) and applies fragmentation penalties. The metric was designed to correlate more strongly with human judgments than BLEU. It remains a useful baseline for MT and NLG evaluation alongside ROUGE and chrF. Implementations exist in NLTK and evaluation toolkits, though newer metrics like BERTScore are increasingly used.

ALTERNATIVE DEFINITIONS

ACL Anthology

METEOR is an automatic metric for machine translation evaluation based on explicit word-to-word matches between the machine translation output and a reference translation.

Defines alignment-based matching at the word level.

Source

Carnegie Mellon University

The METEOR metric is designed to address some of the weaknesses found in BLEU by incorporating stemming, synonyms, and function word filtering.

Emphasizes expanded matching capabilities beyond exact n-grams.

Source

Hugging Face

The meteor score is based on the harmonic mean of unigram precision and recall, with recall weighted higher than precision.

Highlights the precision-recall formulation and recall weighting.

Source

RELATED TERMS

ROUGE Score Machine Translation N-gram Matching Stemming Synonym Matching Paraphrase Matching Translation Evaluation Harmonic Mean

COMPETING TERMS

BLEU chrF

KNOWLEDGE AREAS

AI Data Science Testing & Validation

TARGET AUDIENCE

Data Scientist AI Engineer MLOps Engineer

EXTERNAL SOURCES

• Wikipedia - METEOR
• ACL Anthology - Original METEOR paper
• Carnegie Mellon University - METEOR project page
• Hugging Face - Evaluate METEOR metric docs
• NLTK - NLTK METEOR implementation

TOP QUESTIONS

How does METEOR differ from BLEU?

METEOR aligns words and incorporates stemming, synonyms, and paraphrases, while BLEU focuses on n-gram precision. METEOR typically correlates better with human judgments for MT.

Category: Differences

Why might METEOR be preferred for translation evaluation?

Its alignment and recall weighting capture adequacy better than pure precision metrics. This often yields more human-aligned scores.

Category: Relevancy

What is the fragmentation penalty in METEOR?

It penalizes disordered alignments to reflect fluency and ordering. More fragments imply lower fluency and reduce the final score.

Category: Concept

Is METEOR suitable for summarization evaluation?

It can be used, but ROUGE and newer semantic metrics may be more common. Always validate correlations for your task.

Category: Opinion

What are common pitfalls when using METEOR?

Not enabling language resources (e.g., WordNet) or misconfiguring stemming can underutilize METEOR’s strengths. Comparing across different parameterizations is misleading.

Category: Mistakes

What is a risk of relying solely on METEOR?

It still uses surface-level alignments and may miss deeper semantics. Combining with human evaluation or contextual metrics mitigates this risk.

Category: Risk

How should I report METEOR scores?

Report configuration details (language, stemming, synonym resources) and compare against baselines consistently. Provide confidence intervals when possible.

Category: BestPractices

What are alternatives to METEOR for semantic evaluation?

BERTScore, MoverScore, and BLEURT provide embedding-based semantics. chrF is a strong character-level alternative for morphology-rich languages.

Category: Alternatives

When was METEOR introduced and by whom?

It was introduced in 2005 by Alon Lavie and colleagues at Carnegie Mellon University to improve correlation with human judgments.

Category: History