AnalysisAI ModelsNovember 10, 2026

Raschka: open-weight LLMs drop full attention entirely

Read original source →sebastianraschka.com

In H2 2026, open-weight models split 50/50 between "some full attention" hybrids and designs with zero dense global layers — and no full-attention models remain, per Sebastian Raschka's tracking. Sparse and recurrent layers (Gated DeltaNet, Mamba-2, KDA) cut attention compute and hold fixed-size state, though compressed history complicates exact retrieval.

1 source

More stories today

Open the live feed