Talk: test AI agents like a scientist, not a benchmark
Read original source →youtube.com
Stephanie Jarmak, an ex-astronomer now building evals at Omni Analytics, shows that swapping only the harness moved agent results by 30%. She argues public benchmarks are for shortlisting, not final judgment.
People · Stephanie Jarmak
1 source
More stories today
Coding Agent Index v2.0: Claude Opus 5.5 in Claude Code leads at 61.7
Artificial Analysis·2 hours ago
Grok Bot connects to Calendar, Reminders, Home on iOS and Android
TestingCatalog News·2 hours ago
Rauch highlights Generative UI example built on AI SDK and AI Gateway
Guillermo Rauch·2 hours ago
Tool turns existing agent harnesses into plug-and-play backends
Tom Doerr·2 hours ago
Ultrametric AI explores automating the startup itself
Jude Gomila·2 hours ago
Workshop in SF teaches attendees to train open-weight agents
Sergio Paniego·2 hours ago
Dwarkesh Patel argues AI's bottleneck isn't compute but geometry
Dwarkesh Patel's video argues that geometry, not compute, is the constraint on AI progress. No further details were available in the source snippet.
YouTube·2 hours ago
Talk: San Francisco Compute on GPU clusters and compute markets
Evan Conrad recounts how San Francisco Compute started as an AI lab that subleased its own GPU cluster to survive, then began operating clusters and running a market for them. He traces the chain of problems new labs and inference companies hit, starting with everything being sold out.
YouTube·2 hours ago