Dispersion loss counteracts embedding condensation in small language models

Dispersion loss counteracts embedding condensation in small language models, according to a new research project. The project page provides details and experimental results.