AnalysisAI ModelsOctober 27, 2026

Ai2 benchmark finds LLM novelty judges give inconsistent verdicts

Read original source →huggingface.co

Ai2's "Old Ideas, Novel Problems" tested six judges — GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Opus 4.5 and Opus 4.6 — and found small changes to the evaluation setup produce dramatically different novelty verdicts on the same ideas. Changing just a few words in a prompt shifted a model's judgment.

2 sources

More stories today

Open the live feed