Flash-MSA: Sparse attention kernels for million-token training

Flash-MSA introduces sparse attention kernels to accelerate training of transformers on sequences up to millions of tokens. It reduces compute and memory costs compared to standard attention, enabling longer context training.