MiniMax releases Sparse Attention (MSA) for long-context LLMs

MarkTechPost

The method, built on Grouped Query Attention (GQA), was trained on a 109B-parameter Mixture-of-Experts model with a 3 trillion token budget. It aims to address the quadratic cost of softmax attention for ultra-long contexts in agentic workflows and code reasoning.

Sparse attention

See 4 more sources

Gaussian Mixture Attention: Linear-Time Sequence Mixing via Probabilistic Latent Routing4 days agoYongchao Huang, Hassan Raza

HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization3 days agoZhentao Tan, Wei Chen, Jingyi Shen, Yao Liu, Xu Shen, Yue Wu, Jieping Ye

ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation4 days agoYao Chen, Yinqi Yang, Junyuan Shang, Xiangzhao Hao, Simeng Zhang, Yilong Chen, Tingwen Liu, Shuohuan Wang, Dianhai Yu

MiniMax Sparse Attention (MSA)9 days agopmttyjiDiscuss

MarkTechPost·Asif Razzaq

Jun 17, 7:44 AM