AnalysisDevelopersSeptember 10, 2026

Blog details OpenRouter provider pitfalls for open models

Olly, an iMessage AI assistant, ran ~6M of its 18M messages on open models via OpenRouter. Same DeepSeek V4 Flash 0731 weights scored 90% GPQA/81% TAU on first-party DeepSeek vs 75%/58% on DigitalOcean; Fireworks hit 46% TAU in July.

1 source

More stories today

Open the live feed