Spectral scaling laws of Muon optimizer

Arxiv CS.LG (machine learning)

Paper derives spectral scaling laws for Muon, the orthonormalizer optimizer used in recent open-source LLMs. The analysis reveals how Muon's update rule affects training dynamics across model scales.

muon

See 1 more source

Spectral Scaling Laws of Muon6 days agoGagik Magakyan, Pablo Parrilo, Asuman Ozdaglar

Arxiv CS.LG (machine learning)·Hua Huang

7 days ago