Paper asks if value vectors need residual context

Arxiv CS.CL (NLP/language models)

Investigates whether value vectors in deep transformer layers rely on context from the residual stream. Finds that directing value computation to a separate pathway can maintain or improve performance while reducing computation.

transformerllm

Arxiv CS.CL (NLP/language models)·Muyu HeYuchen LiuQingya HuangLi Zhang

8 days ago