Ai2's BenchMIRT audits what LLM benchmarks actually measure

BenchMIRT analyzes benchmarks question by question, revealing which capabilities they test. On BBQ, a social-bias eval, it found questions distinguished models more by reasoning ability than safety.