Mini Kimi-K3 replica trained for $250 beats GPT-2

The 1.02B-parameter Kimi K3 replica was trained on 5B decontaminated tokens for $250, using 145M active parameters per token — roughly 1/2,000th of Kimi K3's size — and reportedly beats GPT-2 (124M).