Multiple papers probe LLM-as-a-judge reliability and bias

Arxiv CS.CL (NLP/language models)

Papers find LLM judges show run-to-run unreliability across tasks, exhibit language-switching bias, and require psychometric validation to replace human raters. New methods like Metric Match and drift attribution aim to improve evaluation trustworthiness.

Benchmarks

See 4 more sources

Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability6 days agoAlyssa Unell, Natalie Dullerud, Naomi Boneh, Meena Jagadeesan, Tatsu Hashimoto, Nigam Shah, Sanmi Koyejo

Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines6 days agoYitao Li

The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation7 days agoAbel Yagubyan

Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge7 days agoShaojie Yin

Arxiv CS.CL (NLP/language models)·Hiroyasu UsamiKeisuke HaraAyato TsuboiNaohiko Matsuda

Jun 16, 4:00 AM