AnalysisAI Models
29 days ago
LoRA with weight decay regularizes towards base model, not zero
LoRA with weight decay implicitly regularizes adapted weights towards the frozen base model, unlike full finetuning which regularizes towards zero. This means LoRA solves a fundamentally different optimization problem, which practitioners should account for.