Regret Pre-training: Bridging prior and posterior views

Arxiv CS.CL (NLP/language models)

Regret Pre-training leverages future token information during self-supervised training, bridging prior and posterior views. The method improves knowledge grounding compared to standard causal language model pre-training.

trainingllm

Arxiv CS.CL (NLP/language models)·Mingkuan ZhaoXiayu SunWentao HuSuquan ChenJiaxuan LiXiaoyan ZhuXin LaiJiayin Wang

8 days ago