Apple proposes integrated enlarge-and-prune pipeline for LLM pretraining

Apple researchers propose IDEA Prune, an integrated enlarge-and-prune pipeline that combines enlarged model training, pruning, and recovery under a single cosine annealing schedule. Experiments compressing 2.8B models to 1.3B with up to 2T tokens show superior pruned model performance.