CryptanalysisBench measures LLMs' ability to find new attacks

The new benchmark paper "Can LLMs do Cryptanalysis?" tests LLMs on discovering mathematical cryptanalytic attacks — and Anthropic's frontier model actually found new attacks, per Bruce Schneier.