Studies show LLM judges have reliability and bias issues

Multiple arXiv papers analyze LLM-as-a-Judge reliability, revealing bias toward English and inflated scores without reference answers. One large-scale study across 29 tasks found judge agreement often approaches random chance.