Dan Luu argues LLMs make benchmark gaming trivial

Dan Luu says he sees at least one gamed benchmark claim a week — projects touting huge speedups that don't improve real-world performance. He argues LLMs make hacking large benchmark suites trivial, citing Sun's 12x gain on 179.art in SPECfp2000 as an early example, and walks through his regex engine FRE as a case study.