Gigatoken: Rust BPE tokenizer encodes text at 24.
Gigatoken achieves 24.53 GB/s encoding speed on a single machine, up to 989x faster than HuggingFace tokenizers and ~100x faster than Tiktoken. Released under MIT license by Stanford PhD student Marcel Rød, it is written in Rust and available on GitHub.
Featured · Marcel Rød
1 source
More stories today
Engineers discuss raising AI vendor ethics at work
A Reddit thread asks how developers bring up ethical concerns about AI vendors, citing a workplace policy requiring discussion of environmental and social costs before endorsing AI use.
r/ExperiencedDevs·17 minutes ago
Qwen Image 2.0 Pro drops to 1032 on Text-to-Image Arena
Qwen Image 2.0 Pro (2026-04-22) score on the Artificial Analysis Text-to-Image Arena fell from 1234 to 1032, dropping from #16 to #18. The arena is a blind-comparison benchmark where users vote between paired model outputs.
Artificial Analysis Leaderboard·37 minutes ago
HiDream-O1-Image-1.5 drops to #17 on Artificial Analysis Text-to-Image Arena
HiDream-O1-Image-1.5's score on the Artificial Analysis Text-to-Image Arena fell from 1228 to 1032, dropping it from #18 to #17. The arena is a blind-comparison benchmark where users vote between paired model outputs.
Developers by email
Get an email when there's news on Developers
No news that day, no email.
Seedream 4.0 rises to #16 on Artificial Analysis Text-to-Image Arena
ByteDance Seed's Seedream 4.0 rose from #19 to #16 on the Artificial Analysis Text-to-Image Arena, with its score dropping from 1226.3 to 1032.5. The arena is a blind-comparison benchmark where users vote between paired model outputs.
Grok Imagine image quality score drops to 1034 on Artificial Analysis arena
Grok Imagine (SpaceXAI) fell from 1235 to 1034 on the Artificial Analysis Text-to-Image Arena, holding #15. The arena uses blind user votes comparing paired model outputs.
Qwen-Image-3.0 drops to 1078 on Text-to-Image Arena
Qwen-Image-3.0's score on the Artificial Analysis Text-to-Image Arena fell from 1270 to 1078, ranking #14. The arena is a blind-comparison benchmark where users vote between paired model outputs.
Seedream 5.0 Pro drops to 1082 on Text-to-Image Arena
ByteDance Seed's Seedream 5.0 Pro score fell from 1279 to 1082 on the Artificial Analysis Text-to-Image Arena, ranking #13. The drop likely reflects new model releases or capability updates.
Qwen-Image-3.0-Pro drops to 1086 on Text-to-Image Arena
Qwen-Image-3.0-Pro's score on the Artificial Analysis Text-to-Image Arena fell from 1283 to 1086, ranking #12. The arena is a blind-comparison benchmark where users vote between paired model outputs.
