How-ToAI Models
4 hours ago
Lecture on distillation literature from Hinton 2015 to modern methods
Nathan Lambert
@natolambert.bsky.socialA LLN - large language Nathan - (RL, RLHF, society, robotics), athlete, yogi, chef Writes http://interconnects.ai Prev Ai2/Olmo, HuggingFace, Berkeley, and normal places
Nathan Lambert
@natolambert.bsky.social
New lecture for the book! Nominally about synthetic data, but mostly is a walk through of the distillation literature from the Hinton 2015 paper to multi-teach on-policy distillation of today! At 7.4 hours of video in my post-training brain dump and counting :) youtu.be/6nyJ8y8ghsE
·
4 hours ago