Back to AIBriefs
LaunchDevelopers

Perplexity open-sources Unigram tokenizer with 5-6x CPU reduction

Perplexity
@perplexity_ai

Curiosity changes everything. Download our free app on iOS, Mac, Windows, and Android.

San Francisco, CAperplexity.ai/personal-computer
View on X
View on X
Perplexity AI avatar
Perplexity AI
@perplexity_ai

We're open-sourcing the Unigram tokenizer we rebuilt to reduce CPU utilization by 5-6x. Small rerankers and embedders run in single-digit milliseconds on GPU, making CPU tokenization a meaningful share of total latency. https://t.co/QUnHeiho56 https://t.co/Oh29f1lo51

RT @perplexity_ai: We're open-sourcing the Unigram tokenizer we rebuilt to reduce CPU utilization...14 days agoAravind Srinivas
Every millisecond matters. We’re open sourcing the tokenizer we built and deployed on production;...14 days agoAravind Srinivas
Perplexity just open-sourced the tool they use internally to cut their own CPU usage by 5-6x. 🤯...13 days agoVaibhav Sisinty
Perplexity AI Open-Sources Unigram Tokenizer That Achieves 5x Lower p50 Latency Than Hugging Face tokenizers Crate13 days agoAsif Razzaq
·
14 days ago
Perplexity open-sources Unigram tokenizer with 5-6x CPU reduction — AIBriefs