facebookresearch / swe-sweep

SWE-sweep 
 How many bugs can LMs find & fix in large codebases? 
Given a real repository, an agent must discover & repair as many bugs as they can. 
Agents are not given any hint about the type of bug or its location.
## Links
- Website: https://swesweep.com
- Paper: https://swesweep.com/paper
## Quickstart
> [!note]
> The code in this repo is only a thin wrapper around harbor, however see the warning below regarding harbor version
> [!warning]
> * **Harbor version**: Harbor 0.23 does not yet support the separate verifier environments and collect hooks
> used by these tasks. The project therefore pins a compatible revision from Harbor's source
> repository until Harbor 0.24 is released.
> * **Final score**: We build a micro-average of the bugs resolved (equivalent to a weighted average of the
> individual task scores). See evaluation notes.
We recommend uv for managing
Python environments.
 [code] 
Verify your setup:
 [code] 
 Development setup 
 Clone the repository and install the editable package with its dev dependencies:
 [code] 
 Run the test suite (matches CI, which tests on Python 3.12 and 3.13):
 [code] 
 Run the linting/formatting hooks:
 [code] 
## Evaluate a single task
Pass a task name and a unified diff against that task's base commit:
 [code] 
 Raw harbor command 
 [code] 
The command runs the corresponding Harbor task with a small patch-applying agent. Harbor
then evaluates the resulting checkout in the task's separate verifier environment. Results
are written under `jobs/` by default.
 How evaluation works 
Evaluation follows the following pseudo-code:
 [code] 
## Calculate the final score
The benchmark score is calculated as
 [code] 
Summarize one or more directories containing graded Harbor trials:
 [code] 
## Citation
 [code] 
## License
SWE-sweep is licensed under the terms of the license found in LICENSE.