AnalysisAI ModelsNovember 4, 2026

Essay argues no posttraining data commons exists for LLMs

Read original source →openathena.ai

Open Athena's post credits pretraining to a 50-year digital commons — Wikipedia, ArXiv, Stack Exchange, Project Gutenberg and Common Crawl's ~10 petabytes — noting GPT-3 was ~60% Common Crawl-derived and the original Llama mix almost entirely web-sourced. It argues no equivalent shared corpus exists for posttraining.

1 source

More stories today

Open the live feed