Qwen tokenizes 330-line code into 1,609 tokens; Gemma needs 4,258
A LocalLLaMA user ran the same 330-line HTML/JS file through Qwen 35B A3B and Gemma 26B A4B: Qwen tokenized it to 1,609 tokens while Gemma produced 4,258, a gap the poster says helps explain the perceived difference between the models.
1 source
Daily brief
Get tomorrow's AI brief in your inbox
More stories today
- Angelo: Video Edition coming soon for ComfyUI
- DeepZero automates Windows kernel vulnerability research with AI agents
- MiniMax H3 music video clip showcases character consistency and lip-sync
- Compact Magazine essay argues AI will destroy Americans' way of life
- Paper examines the limitations of current AI evaluation methods