Qwen 35B A3B tokenizes code in 1609 tokens vs Gemma 26B's 4258
On a shared 330-line HTML/JS snippet, Qwen 35B A3B produced 1609 tokens while Gemma 26B A4B needed 4258. The r/LocalLLaMA poster says the gap helps explain why Qwen is regarded as better at code.
1 source
Daily brief
Get tomorrow's AI brief in your inbox
More stories today
- China AI Chip Designer Moore Threads Plans Hong Kong Listing
- Ineffable Intelligence founder David Silver pledges company sale to charity
- ChatGPT Finance announced to help you save money
- Evaluation benchmarks released for model performance testing
- Resource explains LLM tokenization, Byte Pair Encoding, and attention math