New joint predict-reconstruct objective for language models

Arxiv CS.CL (NLP/language models)

The paper proposes a self-supervised objective combining masked language modeling and reconstruction to encourage deeper semantic representations. It aims to reduce the surface-form bias of BERT-style models.

self-supervised-learningllm

Arxiv CS.CL (NLP/language models)·Aimen Boukhari

6 days ago