Ben Hylak: Eval advice still written for chatbot era

In a talk, Ben Hylak argues that eval advice is still written for the chatbot era, when answers were known. He reframes evals for agentic systems, noting that switching harnesses can invalidate 80% of a thousand-example eval suite.

Featured · Ben Hylak