AnalysisAI ModelsSeptember 19, 2026

Google engineers find benchmark harnesses silently underdeliver

Ashok Chandrasekar and Jason Kramberger, both at Google, asked a benchmark harness for 200 queries per second and it quietly delivered 38 while still printing results as though it ran 200. The mismatch explains why they kept failing to reproduce published LLM benchmark numbers.

People · Ashok Chandrasekar, Jason Kramberger

1 source

More stories today

Open the live feed
Google engineers find benchmark harnesses silently underdeliver