AppleAnalysisAI ModelsNovember 4, 2026

Apple researchers distill Transformer into pure Mamba model

Read original source →machinelearning.apple.com

A two-stage recipe — Transformer into linearized attention, then into an attention-free Mamba — retains most of Pythia-1B's performance, hitting perplexity 14.01 versus the teacher's 13.86. Gaps appear on tasks relying on in-context recall.

1 source

More stories today

Open the live feed