GAUGE · Classification
Classification.
Pick one label or handler for an input.
classify.email
Classify an internal operations email into one of 18 business categories, with retrieved category examples and similar earlier tasks as context.
Home advantage. gpt-5.4 is the production model on this task: the reference answers are its own outputs, so it is scored against itself. Read its rows, and any tie it is in, with that in mind.
Top of the tied group
11 models, not separated
highest point estimate 89.3 ±6.3, at n = 100
Tied with the best
11 models (18 rows)
within 6.3 of the best
Fastest among tied
gpt-6-luna
median · API round trip
Cheapest API among tied
gpt-6-luna
$0.25 per 1,000 calls
Best local model
pplx-decider:27b
86.0 ±6.9 · tied
| Model | Type | Score ± 95% | n | Repeats | Consistency | Validity | Median latency | $ / 1k calls |
|---|---|---|---|---|---|---|---|---|
| gpt-5.4 | API | 89.3 ±6.3 | 100 | 3 | 96.0% | 100.0% | 1.9 s API | $7.8 |
| claude-sonnet-5-5 · thinking on | API | 88.7 ±6.4 | 100 | 3 | 99.0% | 100.0% | 2.3 s API | $17.8 |
| claude-sonnet-5-5 | API | 88.3 ±6.5 | 100 | 3 | 98.0% | 100.0% | 2.2 s API | $17.5 |
| claude-opus-5-5 · thinking on | API | 88.0 ±6.6 | 100 | 1 | — | 100.0% | 2.8 s API | $36.8 |
| claude-opus-5-5 · minimum thinking | API | 87.0 ±6.8 | 100 | 1 | — | 100.0% | 2.5 s API | $35.5 |
| gpt-6-luna · thinking on | API | 86.7 ±6.8 | 100 | 3 | 92.0% | 100.0% | 2.4 s API | $0.31 |
| claude-haiku-5-5 · thinking on | API | 86.3 ±6.9 | 100 | 3 | 93.0% | 100.0% | 2.3 s API | $1.0 |
| gemini-3.8-flash | API | 86.3 ±6.9 | 100 | 3 | 99.0% | 100.0% | 2.6 s API | $2.9 |
| pplx-decider:27b | Decision | 86.0 ±6.9 | 100 | 1 | — | 100.0% | 9.6 s local | — |
| gemini-3.8-flash · thinking on | API | 85.0 ±7.1 | 100 | 3 | 97.0% | 100.0% | 7.8 s API | $7.6 |
| gemini-4-argon · thinking on | API | 85.0 ±7.1 | 100 | 3 | 98.0% | 99.7% | 13.3 s API | $18.9 |
| qwen3.8:27b-mlx · 27.8B | Local | 85.0 ±7.1 | 100 | 1 | — | 100.0% | 6.7 s local | — |
| gpt-5.4 · thinking on | API | 84.3 ±7.2 | 100 | 3 | 93.0% | 100.0% | 4.4 s API | $11.8 |
| gpt-6-luna | API | 84.3 ±7.2 | 100 | 3 | 94.0% | 100.0% | 1.3 s API | $0.25 |
| gemini-4-argon · minimum thinking | API | 84.0 ±7.3 | 100 | 3 | 98.0% | 100.0% | 7.0 s API | $10.9 |
| gemma4:31b-mlx · 31.7B | Local | 84.0 ±7.3 | 100 | 1 | — | 100.0% | 9.6 s local | — |
| gpt-6.1-terra | API | 83.7 ±7.3 | 100 | 3 | 97.0% | 100.0% | 1.5 s API | $2.3 |
| gpt-6.1-terra · thinking on | API | 83.7 ±7.3 | 100 | 3 | 93.0% | 100.0% | 1.8 s API | $2.5 |
| gpt-6.1-sol · thinking on | API | 82.3 ±7.5 | 100 | 3 | 98.0% | 100.0% | 2.2 s API | $5.1 |
| gpt-6.1-sol · minimum thinking | API | 82.0 ±7.6 | 100 | 3 | 98.0% | 100.0% | 2.0 s API | $4.8 |
| gemma4:12b-mlx · 12.4B | Local | 81.0 ±7.7 | 100 | 1 | — | 100.0% | 3.5 s local | — |
| bonsai-2:27b · 26.9B | Local | 80.0 ±7.8 | 100 | 1 | — | 100.0% | 13.1 s local | — |
| claude-haiku-5-5 | API | 72.0 ±8.7 | 100 | 3 | 94.0% | 100.0% | 1.2 s API | $0.87 |
| ministral-3:14b · 13.9B | Local | 69.0 ±8.9 | 100 | 1 | — | 100.0% | 10.3 s local | — |
| d3:27b | Decision | 59.0 ±9.5 | 100 | 1 | — | 70.0% | 24.3 s local | — |
Amber edge and bar: tied with the best (score ≥ best − its 95% margin, harness rule). Only rows with the task's full n are compared.
route.question
Route a user question in a financial-products assistant to one of 3 handlers (documentation, products, other).
Top of the tied group
4 models, not separated
highest point estimate 98.0 ±3.7, at n = 100
Tied with the best
4 models (6 rows)
within 3.7 of the best
Fastest among tied
gpt-6-luna
median · API round trip
Cheapest API among tied
gpt-6-luna
$0.090 per 1,000 calls
Best local model
pplx-decider:27b
95.0 ±4.8 · tied
| Model | Type | Score ± 95% | n | Repeats | Consistency | Validity | Median latency | $ / 1k calls |
|---|---|---|---|---|---|---|---|---|
| gpt-6.1-sol · minimum thinking | API | 98.0 ±3.7 | 100 | 3 | 100.0% | 100.0% | 1.8 s API | $1.7 |
| gpt-6.1-sol · thinking on | API | 97.3 ±4.0 | 100 | 3 | 99.0% | 100.0% | 2.1 s API | $2.0 |
| pplx-decider:27b | Decision | 95.0 ±4.8 | 100 | 1 | — | 100.0% | 2.7 s local | — |
| gpt-6-luna | API | 95.0 ±4.8 | 100 | 3 | 98.0% | 100.0% | 1.1 s API | $0.090 |
| gpt-6.1-terra | API | 95.0 ±4.8 | 100 | 3 | 96.0% | 100.0% | 1.3 s API | $0.77 |
| gpt-6.1-terra · thinking on | API | 94.7 ±4.9 | 100 | 3 | 95.0% | 100.0% | 1.6 s API | $0.99 |
| d3:27b | Decision | 94.0 ±5.1 | 100 | 1 | — | 100.0% | 3.0 s local | — |
| bonsai-2:27b · 26.9B | Local | 94.0 ±5.1 | 100 | 1 | — | 100.0% | 4.9 s local | — |
| qwen3.8:27b-mlx · 27.8B | Local | 94.0 ±5.1 | 100 | 1 | — | 100.0% | 3.1 s local | — |
| claude-opus-5-5 · minimum thinking | API | 93.3 ±5.3 | 100 | 3 | 99.0% | 100.0% | 1.7 s API | $6.4 |
| claude-opus-5-5 · thinking on | API | 93.3 ±5.3 | 100 | 3 | 98.0% | 100.0% | 1.8 s API | $7.3 |
| gemini-4-argon · thinking on | API | 93.3 ±5.3 | 100 | 3 | 98.0% | 100.0% | 9.0 s API | $7.9 |
| gpt-5.4 · thinking on | API | 93.3 ±5.3 | 100 | 3 | 96.0% | 100.0% | 3.0 s API | $7.2 |
| gpt-6-luna · thinking on | API | 93.3 ±5.3 | 100 | 3 | 94.0% | 100.0% | 1.7 s API | $0.13 |
| claude-sonnet-5-5 · thinking on | API | 92.3 ±5.6 | 100 | 3 | 94.0% | 100.0% | 1.7 s API | $4.0 |
| gemini-3.8-flash | API | 92.3 ±5.6 | 100 | 3 | 95.0% | 100.0% | 2.1 s API | $1.2 |
| gpt-5.4 | API | 92.3 ±5.6 | 100 | 3 | 98.0% | 100.0% | 1.5 s API | $4.6 |
| gemini-4-argon · minimum thinking | API | 92.0 ±5.7 | 100 | 3 | 98.0% | 100.0% | 6.4 s API | $4.0 |
| gemma4:12b-mlx · 12.4B | Local | 92.0 ±5.7 | 100 | 1 | — | 100.0% | 1.3 s local | — |
| claude-haiku-5-5 · thinking on | API | 91.7 ±5.7 | 100 | 3 | 91.0% | 100.0% | 1.1 s API | $0.21 |
| gemini-3.8-flash · thinking on | API | 91.0 ±5.9 | 100 | 3 | 98.0% | 100.0% | 5.0 s API | $3.9 |
| gemma4:31b-mlx · 31.7B | Local | 91.0 ±5.9 | 100 | 1 | — | 100.0% | 2.5 s local | — |
| claude-sonnet-5-5 | API | 90.3 ±6.1 | 100 | 3 | 96.0% | 100.0% | 1.9 s API | $3.3 |
| claude-haiku-5-5 | API | 88.7 ±6.4 | 100 | 3 | 97.0% | 100.0% | 1.0 s API | $0.17 |
| ministral-3:14b · 13.9B | Local | 81.0 ±7.7 | 100 | 1 | — | 100.0% | 3.1 s local | — |
Amber edge and bar: tied with the best (score ≥ best − its 95% margin, harness rule). Only rows with the task's full n are compared.