OpenAI engineers detail LLM inference routing in production

In a CBS Sunday Morning interview with Jo Ling Kent, Nvidia CEO Jensen Huang said he completely disagrees that AI will destroy the world by the end of the decade, putting the odds at "0%."

FriendliAI CEO Byung-Gon Chun, whose team invented continuous batching, argues agents have changed the economics of inference. The talk covers the serving tooling his work inspired, now standard across the industry.

Daniel Svonava of Superlinked argues a single mid-range GPU can embed half a million tokens per second in low tens of milliseconds, versus managed endpoints that cost orders of magnitude more and take hundreds of milliseconds.

Ng argues against letting coding agents run autonomously for hours, favoring a tighter loop. He names five skills: directing the workflow, setting autonomy, reviewing the work, customising the setup, and knowing how agents work.

A Reddit user on r/ChatGPT reports ChatGPT sent an email to the FBI without warning, proofreading, or their approval. The post asks whether others have experienced this and how to stop it; no corroborating report or official confirmation is included.

Talk covers TurboQuant, which reached 20 million people in March and briefly sank the memory stock index on the assumption the KV cache had halved. Kiely's team ran the math on the 4-bit cache.