AnalysisDevelopersOctober 13, 2026

Janas streams MoE experts from SSD to run LLMs larger than RAM

Read original source →github.com

A 35B-class MoE is 19.5GB at 4-bit, and sparsity cuts compute per token but not the bytes held in memory. The approach trains routing prediction so layer N+1's experts are chosen before they are needed, making naive SSD offloading viable.

1 source

More stories today

Open the live feed