DataTerminology.comNEW
A benchmark of programming problems with unit tests to evaluate LLM code generation via functional correctness metrics like pass@k.
DESCRIPTION
HumanEval is a widely used benchmark for assessing code generation by large language models through unit-tested programming problems. Models generate functions from docstrings or prompts, and success is measured by passing hidden tests. It popularized the pass@k metric to account for stochastic sampling. While compact, it correlates with model capability trends and is complemented by larger suites like MBPP and APPS. Extensions (e.g., EvalPlus) address test adequacy and robustness.
ALTERNATIVE DEFINITIONS
OpenAI
This repository contains the HumanEval dataset and evaluation harness for evaluating the functional correctness of language models for code completion.
Defines HumanEval as both dataset and evaluation harness, emphasizing functional correctness.
arXiv
We evaluate Codex on HumanEval, a benchmark of 164 hand-written programming problems with unit tests that assess functional correctness.
Specifies benchmark size, hand-written nature, and unit-test-based evaluation.
RELATED TERMS
Program Synthesis Pass@k BigCodeBench Unit Testing Prompt Engineering LLM Evaluation Zero-shot Learning
COMPETING TERMS
MBPP APPS
KNOWLEDGE AREAS
AI Data Science Testing & Validation Programming Languages
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer Software Developer AI Product Manager
EXTERNAL SOURCES
- OpenAI - Official dataset and evaluation harness
- Papers with Code - Community overview and results
- arXiv - EvalPlus improving test coverage
TOP QUESTIONS
Why is HumanEval widely used for code generation evaluation?
It offers a simple, reproducible setup with unit tests and a standard pass@k metric, enabling apples-to-apples comparisons across models.
Category: Relevancy
What does pass@k mean in HumanEval?
It estimates the probability that at least one of k samples passes all tests, accounting for stochastic generation.
Category: Example
What are limitations of HumanEval?
Dataset size is small, tests may be insufficient, and it doesn’t cover complex, multi-file projects or security aspects.
Category: Risks
How should I run HumanEval correctly?
Use the official harness, avoid prompt leakage, control sampling parameters, and report pass@k with confidence intervals.
Category: BestPractices
Which benchmarks compete with HumanEval?
MBPP and APPS offer broader problem sets and alternative difficulty distributions.
Category: Alternatives
How does HumanEval relate to real-world coding tasks?
It approximates function-level tasks but omits tooling, dependencies, and long-horizon reasoning common in production development.
Category: Misunderstanding
What is a hot topic around HumanEval?
Improving test coverage (EvalPlus), prompt decontamination, and coding tool integration for realistic evaluation.
Category: Topic
Can HumanEval be used for regression testing during model finetuning?
Yes, but complement it with larger benchmarks and domain-specific tasks to avoid overfitting.
Category: HowTo
How do I avoid contamination when reporting HumanEval scores?
Verify training data, use held-out prompts, and disclose potential overlaps and mitigation steps.
Category: BestPractices