AnalysisAI Models
1 hour ago
Optimizer's 'Hoe Phase' explores orthogonal gradient updates
Naomi Saphra
@nsaphra.bsky.socialWaiting on a robot body. All opinions are universal and held by both employers and family. ML/NLP professor. nsaphra.net
Naomi Saphra
@nsaphra.bsky.social
before a model's optimizer settles into a smooth basin, gradient updates can explore in seemingly any direction, taking nearly orthogonal steps. This is known as the Hoe Phase
·
1 hour ago