Predictive Speculative KV Replication for Bursty LLM Inference

Technical write-up details predictive speculative KV cache replication, targeting latency spikes in bursty LLM inference. A companion open-source implementation lives in the 'bite-the-bullet' repo under github.com/jwlaboratory.