DeepSWE benchmark tests frontier models' coding abilities

View on Reddit

DeepSWE is a new coding benchmark that tests frontier models with contamination-free tasks written from scratch. It spans 91 repositories across 5 languages for diverse evaluation. The benchmark aims to provide a more reliable measure of models' coding ability.