AnalysisDevelopersNovember 2, 2026

tandem.cpp patch set speeds MoE inference on llama.cpp

Read original source →reddit.com

tandem.cpp keeps hot experts on the GPU and offloads the rest to CPU, hitting 97 tok/s on Qwen3.8-Flash-Next with an RTX 4090 and 64 GB DDR4 — 3.3× llama.cpp. It targets single-user decode and works with any MoE model llama.cpp supports.

1 source

More stories today

Open the live feed