AmazonLaunchDevelopersNovember 17, 2026

SageMaker AI real-time endpoints add KV cache-aware routing

Read original source →aws.amazon.com

The new routing strategy reads cache state reported by the model container and sends each request to an instance holding a long cached prompt match that isn't too busy. It targets real-time endpoints on Amazon SageMaker AI.

1 source

More stories today

Open the live feed