OpenAI identifies reliability issues in SWE-Bench Pro benchmark

The analysis raises concerns about the benchmark's accuracy and reliability for evaluating AI model coding abilities. OpenAI details how the benchmark may conflate signal with noise.