Talk: agent harnesses matter more as models get smarter

Irene Solaiman said at TechCrunch Disrupt that evaluations are "an evolving science" the industry doesn't know how to meaningfully compare, and that reproducibility hinges on open weights. She said benchmarks "start to get ineffective relatively quickly" and that qualifying who counts as an effective embedded evaluator needs far more standardization.

OpenAI says its latest model advances strengthen building and migrating software in low-resource languages like COBOL, Fortran, and Visual Basic, which power bank, government, and business systems. It added new evaluations covering software engineering, SQLite, and COBOL migration.

Guardian Agents is part of Project Perception, Microsoft's June product that launched red-team, blue-team and green-team agents. CEO Satya Nadella warns enterprises must not trust model providers' safety assurances: "we need to separate the supply of intelligence from the authority over it."

Agent Memory on Agent Bricks, in beta, gives agents durable long-term memory stored on Lakebase inside the customer's Databricks workspace, so each conversation starts with what the agent learned previously.

Dan Ives of Yorkville, Ives and Company says upcoming earnings from major tech companies will be an important test of confidence in AI. He also discusses giving investors a chance to invest in private AI companies.