Qwen tokenizes same code in 1,609 tokens vs Gemma's 4,258
A Reddit user found that tokenizing the same 330-line HTML/JS snippet yields 1,609 tokens for Qwen 35B A3B vs 4,258 tokens for Gemma 26B A4B, suggesting more compact tokenization may explain Qwen's better coding performance.
1 source
Daily brief
Get tomorrow's AI brief in your inbox
More stories today
- GOP panics over Big Tech ties as Trump shifts on AI regulation
- Ethan Mollick: Claude's skill creator beats ChatGPT for reusable skills
- Aident Loadout gives agents 27,000+ tools and logs every action
- Etched gains sizable fan base for AI inferencing computers
- Corbell generates technical specs from repository knowledge graphs