Mini Kimi K3 replica trained for $250 beats GPT-2 (124M)

A 1.02B-parameter miniature replica of Kimi K3, trained on 5 billion decontaminated tokens for $250, reportedly beats GPT-2 (124M). The model has 145M active parameters per token and is roughly 1/2000th of Kimi K3's size.