DataTerminology.comNEW
A benchmark evaluating models on real-world software engineering tasks by resolving repository issues through code changes.
DESCRIPTION
SWE-Bench assesses whether models can fix real GitHub issues by modifying multi-file repositories and passing tests. It goes beyond toy problems to measure repository-level reasoning, dependency management, and build/test workflows. Tasks include reproducing failures, editing code, and verifying with unit tests. It supports evaluation of agentic coding systems and tool-integrated approaches. Scores reflect end-to-end problem solving under realistic constraints.
ALTERNATIVE DEFINITIONS
arXiv
We introduce SWE-bench, a benchmark for evaluating large language models’ ability to resolve real-world GitHub issues by making code changes that pass unit tests.
Highlights real-world issues, repository edits, and unit-test validation.
SWE-Bench
SWE-bench evaluates models on repository-level software engineering tasks constructed from real GitHub issues, requiring code modifications validated by tests.
Stresses repository-level granularity and test-driven verification.
RELATED TERMS
Program Repair Repository-Level Tasks Automated Software Engineering Pull Request GitHub Issues Unit Tests Agentic Coding Function Calling
COMPETING TERMS
HumanEval CodeXGLUE
KNOWLEDGE AREAS
Software Engineering AI Testing & Validation MLOps Data Science
TARGET AUDIENCE
Software Developer AI Engineer MLOps Engineer Data Scientist DevOps Engineer Software Architect
EXTERNAL SOURCES
- arXiv - Original SWE-bench paper and setup
- SWE-Bench - Benchmark website, tasks, and results
- Princeton NLP - Benchmark code and dataset repo
TOP QUESTIONS
How does SWE-Bench differ from coding benchmarks like HumanEval?
It evaluates repository-level issue resolution with multi-file changes and unit tests, not single-function synthesis.
Category: Comparison
What capabilities are necessary to do well on SWE-Bench?
Understanding project structure, reading failing tests, making targeted edits, and running build/test workflows are essential.
Category: Requirements
Can agents use tools during SWE-Bench evaluation?
Yes, many approaches integrate tools like search, editors, and test runners. Tool orchestration often boosts success.
Category: Explanation
What are common failure modes on SWE-Bench?
Models struggle with build environment setup, understanding dependencies, and making minimal, correct changes.
Category: Mistakes
Why is SWE-Bench considered realistic?
Tasks derive from actual GitHub issues and require code changes validated by existing unit tests.
Category: Relevancy
How should organizations apply SWE-Bench results?
Use them as a baseline for repository-level reasoning, then test on your codebases and CI pipelines.
Category: HowTo
Does SWE-Bench cover security or performance bugs?
It focuses on functional correctness via tests; security and performance need separate evaluations.
Category: Limitations
How is success measured on SWE-Bench?
By passing the relevant unit tests after applying proposed code changes to the repository.
Category: Testing
What’s a hot topic around SWE-Bench?
Agentic coding workflows and retrieval-augmented reasoning over codebases are active research areas.
Category: Topic