Reddit post: KV cache memory costs challenge softmax attention

View on Reddit

The post argues that the memory bottleneck from KV cache is a critical issue for transformer inference, pointing to rising DDR5 prices as evidence. It suggests softmax attention's cache demands are becoming too expensive.