AnalysisAI ModelsNovember 17, 2026

Raschka: Some LLM pretraining recipes drop LR decay

Read original source →sebastianraschka.com

Sebastian Raschka notes a shift toward constant learning rates plus averaging the last N checkpoints, used by Kolibri-1 and DatologyAI's models. Cosine decay remains standard for Llama 3, OLMo 2/3, GLM-4.5/5, Kimi K3 and Phi-4; WSD for DeepSeek V3/V4, Kimi K2 and MiniMax M2.

1 source

More stories today

Open the live feed