Sliding-window attention beats linear attention on long-context tasks

A new arXiv paper finds sliding window attention with sinks outperforms post-trained linear attention on long-context tasks without retraining, offering cheaper and more reliable inference. Tested on Needle-in-a-Haystack and BABILong.