Ai2's BenchMIRT audits what LLM benchmarks actually measure

Runpod staff engineer Dean Quiñanola presents Runpod Flash, which runs Python on cloud GPUs as if the GPU were local, eliminating Dockerfiles and image builds. The build-push-run-debug loop it replaces can take 15 minutes per iteration.

A developer had Opus 5.5 build a headless drawing tool first, then used it to produce an SVG cutout of a PS5 controller for a mini golf game. The workflow renders each drawing as a script, with the model inspecting output and iterating on fixes.

Conference session by Michael Forrester (Accenture) and Whitney Lee (Datadog) walks through a deliberately permissive burrito-ordering assistant that deploys an external container image and leaks a secret recipe from its Kubernetes environment. Audience is invited to push the bot beyond ordering food, then repeat the attacks with infrastructure and AI tooling.

Wolfram Ravenwolf's talk shows an agent earning a perfect score repairing a broken git repository, yet staying off the default leaderboard because it tested only one task. The pipeline ties each score to the conditions that produced it.

Two Minute Papers walks through DeepSeek's OCR paper, which the video credits with solving an AI memory bottleneck. The paper code is published at github.com/deepseek-ai/DeepSeek-OCR.

Talk by Joshua Mo, lead developer relations engineer at Venice AI and former lead maintainer of the Rust AI framework Rig, argues a private model doesn't make a private product — leaks happen in the surrounding stack. Cites user churn after a single breach as the driver for storing less data.