Paper questions whether agent benchmarks measure true capability

The paper argues that benchmark scores support capability claims only when the evaluation protocol keeps the intended capability necessary for success. It examines agent benchmarks for repository editing, web research, terminal use, and long-horizon interaction.