Microsoft Research proposes Next-Latent Prediction Transformers

View on Reddit

Microsoft Research proposes Next-Latent Prediction Transformers, where models learn to predict their own next latent state instead of the next token. This addresses the myopia of next-token prediction. The preprint is available on Microsoft Research.