LaunchDevelopersNovember 7, 2026

Buoyant releases Kinference, an open source inference control plane for Kubernetes and…

Read original source →marktechpost.com

Kinference v0.1.0-alpha.7, written in Rust under Apache 2.0, sits in front of vLLM workers to apply routing policies, token rate limits, KV-cache-aware load balancing and per-tenant metrics via 5 Kubernetes CRDs. Buoyant's benchmark on 3 L40S GPUs per pool cut median TTFT from 6.88s to 0.31s on shared-prefix traffic, but showed no gain (6.92s vs 6.83s) without shared prefixes.

1 source

More stories today

Open the live feed