Frontier models catch only 50% of vulnerabilities on repeated runs

In a talk, Snyk's Manoj Nair shows that even unreleased frontier models detect a given vulnerability only 50% of the time across five attempts. Against a deterministic checker, they find at most 75% of issues with a 40% F1 score, highlighting architectural challenges for agentic security.

Featured · Manoj Nair