AnalysisAI Models
27 days ago
Long-context efficiency drives new LLM architecture designs
Raschka's analysis covers KV sharing in Gemma 4, layer-wise attention budgeting in Laguna XS.2, compressed convolutional attention in ZAYA1-8B, and mHC in DeepSeek V4. The common theme is reducing KV-cache size and attention cost for longer reasoning and agent tasks. The article focuses on transformer block modifications and memory optimization.