Best open-weight model for a support agent
Compare with other modelsThe open-weight models score well on several of the dimensions we test. DeepSeek V4.1 Flash scores 97 on tool use, while GLM 5.3 Flash and Qwen3.8 27B score 95 and 94 on task completion. But the share of conversations with no unsupported claims is still only 48% at best.
Start with DeepSeek. It scores 85 on quality at about $0.013 per conversation. Qwen offers similar quality, but costs almost twice as much. Gemma 4 31B is cheaper and faster than both, at a lower quality score of 77.
Scroll horizontally to see all models.
| Metric | DeepSeek Sep 15, 2026 | Alibaba Oct 22, 2026 | Zhipu Sep 10, 2026 | Google Sep 8, 2026 |
|---|---|---|---|---|
| Overview | ||||
| Quality score | 85 #2 of 17 | 82 #6 of 17 | 81 #7 of 17 | 77 #10 of 17 |
| Median reply | 5.1s Served by Phala | 5.1s Served by Wafer | 6.4s Served by Together | 2.5s Best among selected models Served by CoreWeave |
| Cost per conversation | $0.013 | $0.024 | $0.011 | $0.0079 Best among selected models |
| Quality scores | ||||
| Tool use | 97 Best among selected models | 89 | 96 | 69 |
| Task completion | 89 | 94 | 95 Best among selected models | 79 |
| Context retention | 81 | 85 Best among selected models | 75 | 79 |
| Grounding | 93 Best among selected models | 92 | 90 | 92 |
| Safety | 96 Best among selected models | 90 | 94 | 90 |
| Hallucinations % of conversations with no unsupported claims detected | 48% Best among selected models | 33% | 24% | 48% Best among selected models |
| Running stats | ||||
| p95 reply | 19.6s | 22.1s | 34.9s | 16.2s Best among selected models |
| Consistency % of repeat runs ended the same way | 85% Best among selected models | 79% | 82% | 82% |
| Output per turn | 450 tokens, median | 693 tokens, median | 600 tokens, median | 88 tokens, median |
Reply times cover the model call only. Cost estimates use token usage at list prices.
This whole report is one Voxli workspace: simulated customers, assertion checks, and a frozen, versioned test set that reruns when new models ship.
Test set v3f-2026-09 · edition 2026-09-08