Study questions necessity of QKV projections in Transformer attention

Arxiv CS.LG (machine learning)

ArXiv paper systematically evaluates variants of Transformer attention that omit query, key, or value projections. Results show that dropping the value projection often has minimal impact on performance across tasks.

transformer

Arxiv CS.LG (machine learning)·Ali KayyamAnusha Madan GopalM Anthony Lewis

7 days ago