Vercel explains how to build evals for AI agents
Read original source →vercel.com
Vercel's guide proposes an "agentic testing pyramid" grouped by LLM calls per test: deterministic tests (0), one-turn evals (1), and E2E evals (2+ across a whole task). It argues cheap, fast tests should make up the bulk of a suite, since LLM output varies between runs and one manual test can't confirm a fix.
2 sources
More stories today
AMD's hrx-loom stack builds Strix Halo kernels for top-51 LLMs
AMD AI Group's experimental hrx-loom bundles a compiler, runtime, and serving tools to build, optimize, serve, and share kernels. An AMD engineer (jfowers_amd) reports using it to produce speedy Strix Halo kernels for the top-51 LLMs.
r/LocalLLaMA·1 hour ago
Mystery AI Hype Theater 3000 tackles doomer narratives and data center lobbying
Emily M. Bender·1 hour agoGS Caltex runs production MLOps on SageMaker Canvas and Pipelines
The South Korean energy and chemical company splits model development and operations between process engineers and dedicated ops staff, connecting engineer-built models to production workflows via SageMaker Pipelines.
AWS AI Blog·1 hour ago
Claude Projects adds parallel threads and a proactive coordinator
Projects in Claude Code now let Claude coordinate tasks across parallel threads, with a more proactive coordinator answering questions in project chat and reporting thread status. The redesign is in public beta on Pro and Max, rolling out in stages with a waitlist.
claude.com·1 hour ago
Claude adds read-only shareable links for individual threads
Claude Developers·1 hour ago
LayerShop adds generative AI fill and Minimax H3 Image support
Open-source Photoshop alternative LayerShop now supports generative fill for adding and removing objects using local models. The latest version also adds support for Minimax H3 Image.
r/StableDiffusion·2 hours ago
AWS shows routing coding-agent requests across models on Bedrock via NVIDIA NeMo…
AWS post walks through using NVIDIA NeMo Switchyard on Amazon Bedrock to send each coding-agent request to the right model, noting a single task can take a few model calls or dozens. It frames cost around the task rather than one call.
Morgan Stanley's Shalett stays bullish on AI capex despite turmoil
Morgan Stanley Wealth Management's Lisa Shalett says AI infrastructure capital spending remains the bull market's main driver, even as Nvidia leadership turmoil and Democratic midterm wins stoke AI-boom concerns.
Bloomberg Technology·2 hours ago
