Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent
SWE-bench-style datasets still contain unclear tasks and weak test coverage. ExLabel uses a self-evolving experience bank to label clarity and coverage across eight languages.
The claim
ExLabel labels SWE-bench-style datasets with experience that improves after every instance.
The problem
Coding benchmarks need clear tasks and strong test coverage.
Static rubrics are too general and cause false positives.
Manual annotation is costly and slow.
How it works
ExLabel keeps an experience bank across assessed instances.
The bank distills stable yet adaptive judgment criteria.
Relevant past experience guides each new assessment.
It judges task clarity and test coverage separately.
SWE-TQ provides cleaned human labels in eight languages.
Figures
What changes
ExLabel reaches the best F1 in every setting tested.
It lifts clarity scores by 44.2% and 49.9% on Python and multilingual data.
Models trained on its filtered data resolve 66.2% of SWE-bench Verified issues.
Limits
Experiments use three backbone language models.
The benchmark is built from SWE-bench and Multi-SWE-bench annotations.
The abstract reports no training cost for the agent.
Source · arXiv