GAUGE · All models
Every model, one board.
Frontier APIs, local open-weight models and small decision models, task by task, in every thinking mode that was run.
| Model | Type | Score ± 95% | n | Repeats | Consistency | Validity | Median latency | $ / 1k calls |
|---|---|---|---|---|---|---|---|---|
| classify.email · accuracy · n = 100 | ||||||||
| gpt-5.4 | API | 89.3 ±6.3 | 100 | 3 | 96.0% | 100.0% | 1.9 s | $7.8 |
| claude-sonnet-5-5 | API | 88.3 ±6.5 | 100 | 3 | 98.0% | 100.0% | 2.2 s | $17.5 |
| claude-opus-5-5 · minimum thinking | API | 87.0 ±6.8 | 100 | 1 | — | 100.0% | 2.5 s | $35.5 |
| gemini-3.8-flash | API | 86.3 ±6.9 | 100 | 3 | 99.0% | 100.0% | 2.6 s | $2.9 |
| pplx-decider:27b | Decision | 86.0 ±6.9 | 100 | 1 | — | 100.0% | 9.6 s | — |
| qwen3.8:27b-mlx | Local | 85.0 ±7.1 | 100 | 1 | — | 100.0% | 6.7 s | — |
| gpt-6-luna | API | 84.3 ±7.2 | 100 | 3 | 94.0% | 100.0% | 1.3 s | $0.25 |
| gemini-4-argon · minimum thinking | API | 84.0 ±7.3 | 100 | 3 | 98.0% | 100.0% | 7.0 s | $10.9 |
| gemma4:31b-mlx | Local | 84.0 ±7.3 | 100 | 1 | — | 100.0% | 9.6 s | — |
| gpt-6.1-terra | API | 83.7 ±7.3 | 100 | 3 | 97.0% | 100.0% | 1.5 s | $2.3 |
| gpt-6.1-sol · minimum thinking | API | 82.0 ±7.6 | 100 | 3 | 98.0% | 100.0% | 2.0 s | $4.8 |
| gemma4:12b-mlx | Local | 81.0 ±7.7 | 100 | 1 | — | 100.0% | 3.5 s | — |
| bonsai-2:27b | Local | 80.0 ±7.8 | 100 | 1 | — | 100.0% | 13.1 s | — |
| claude-haiku-5-5 | API | 72.0 ±8.7 | 100 | 3 | 94.0% | 100.0% | 1.2 s | $0.87 |
| ministral-3:14b | Local | 69.0 ±8.9 | 100 | 1 | — | 100.0% | 10.3 s | — |
| d3:27b | Decision | 59.0 ±9.5 | 100 | 1 | — | 70.0% | 24.3 s | — |
| claude-sonnet-5-5 · thinking on | API | 88.7 ±6.4 | 100 | 3 | 99.0% | 100.0% | 2.3 s | $17.8 |
| claude-opus-5-5 · thinking on | API | 88.0 ±6.6 | 100 | 1 | — | 100.0% | 2.8 s | $36.8 |
| gpt-6-luna · thinking on | API | 86.7 ±6.8 | 100 | 3 | 92.0% | 100.0% | 2.4 s | $0.31 |
| claude-haiku-5-5 · thinking on | API | 86.3 ±6.9 | 100 | 3 | 93.0% | 100.0% | 2.3 s | $1.0 |
| gemini-3.8-flash · thinking on | API | 85.0 ±7.1 | 100 | 3 | 97.0% | 100.0% | 7.8 s | $7.6 |
| gemini-4-argon · thinking on | API | 85.0 ±7.1 | 100 | 3 | 98.0% | 99.7% | 13.3 s | $18.9 |
| gpt-5.4 · thinking on | API | 84.3 ±7.2 | 100 | 3 | 93.0% | 100.0% | 4.4 s | $11.8 |
| gpt-6.1-terra · thinking on | API | 83.7 ±7.3 | 100 | 3 | 93.0% | 100.0% | 1.8 s | $2.5 |
| gpt-6.1-sol · thinking on | API | 82.3 ±7.5 | 100 | 3 | 98.0% | 100.0% | 2.2 s | $5.1 |
| embed.example_retrieval · recall@5 · n = 201 | ||||||||
| snowflake-arctic-embed2 | Local | 38.2 ±6.7 | 201 | — | — | — | 4.8 s total | — |
| nomic-embed-text-v2-moe | Local | 37.3 ±6.6 | 201 | — | — | — | 4 s total | — |
| text-embedding-3-large | API | 36.8 ±6.6 | 201 | — | — | — | 4.4 s total | $0.0035 |
| text-embedding-3-small | API | 34.7 ±6.5 | 201 | — | — | — | 3 s total | $0.00053 |
| qwen3-embedding:0.6b | Local | 34.6 ±6.5 | 201 | — | — | — | 6.4 s total | — |
| qwen3-embedding:8b | Local | 32.9 ±6.5 | 201 | — | — | — | 26.9 s total | — |
| granite-embedding:278m | Local | 30.3 ±6.3 | 201 | — | — | — | 3.7 s total | — |
| bge-m3 | Local | 29.9 ±6.3 | 201 | — | — | — | 4.6 s total | — |
| embeddinggemma-2:570m | Local | 29.4 ±6.3 | 201 | — | — | — | 6.1 s total | — |
| embeddinggemma-2:740m | Local | 29.4 ±6.3 | 201 | — | — | — | 5.7 s total | — |
| gemini-embedding-001 | API | 29.2 ±6.3 | 201 | — | — | — | 6.8 s total | $0.0032 |
| gemini-embedding-2 | API | 28.7 ±6.2 | 201 | — | — | — | 7.7 s total | $0.0058 |
| paraphrase-multilingual | Local | 26.9 ±6.1 | 201 | — | — | — | 3.9 s total | — |
| embed.task_retrieval · recall@5 · n = 100 | ||||||||
| text-embedding-3-large | API | 24.8 ±8.4 | 100 | — | — | — | 3.3 s total | $0.072 |
| nomic-embed-text-v2-moe | Local | 24.2 ±8.3 | 100 | — | — | — | 3.3 s total | — |
| bge-m3 | Local | 22.6 ±8.2 | 100 | — | — | — | 5.2 s total | — |
| qwen3-embedding:8b | Local | 22.0 ±8.1 | 100 | — | — | — | 66 s total | — |
| gemini-embedding-2 | API | 22.0 ±8.1 | 100 | — | — | — | 4.8 s total | $0.084 |
| text-embedding-3-small | API | 21.2 ±8.0 | 100 | — | — | — | 3.1 s total | $0.011 |
| embeddinggemma-2:570m | Local | 21.0 ±8.0 | 100 | — | — | — | 5.9 s total | — |
| embeddinggemma-2:740m | Local | 21.0 ±8.0 | 100 | — | — | — | 5.8 s total | — |
| qwen3-embedding:0.6b | Local | 20.4 ±7.9 | 100 | — | — | — | 8.4 s total | — |
| paraphrase-multilingual | Local | 19.8 ±7.8 | 100 | — | — | — | 2.6 s total | — |
| snowflake-arctic-embed2 | Local | 19.4 ±7.8 | 100 | — | — | — | 5 s total | — |
| granite-embedding:278m | Local | 18.4 ±7.6 | 100 | — | — | — | 2.9 s total | — |
| gemini-embedding-001 | API | 18.4 ±7.6 | 100 | — | — | — | 4.5 s total | $0.062 |
| match.legal_case · accuracy · n = 100 | ||||||||
| gpt-6.1-terra | API | 100.0 ±2.6 | 100 | 1 | — | 100.0% | 2.3 s | $32.3 |
| gpt-6.1-sol · minimum thinking | API | 99.0 ±3.2 | 100 | 1 | — | 100.0% | 3.2 s | $65.0 |
| gemini-4-argon · minimum thinking | API | 96.0 ±4.5 | 100 | 1 | — | 100.0% | 8.0 s | $71.6 |
| gemini-3.8-flash | API | 93.0 ±5.4 | 100 | 1 | — | 100.0% | 2.8 s | $26.8 |
| gpt-5.4 | API | 93.0 ±5.4 | 100 | 1 | — | 100.0% | 3.4 s | $85.4 |
| claude-opus-5-5 · minimum thinking | API | 90.0 ±6.1 | 100 | 1 | — | 100.0% | 2.9 s | $191.4 |
| gpt-6-luna | API | 86.0 ±6.9 | 100 | 1 | — | 100.0% | 1.9 s | $3.2 |
| bonsai-2:27b | Local | 85.0 ±16.3 | 20 | 1 | — | 100.0% | 35.6 s | — |
| gemma4:31b-mlx | Local | 85.0 ±16.3 | 20 | 1 | — | 100.0% | 30.6 s | — |
| qwen3.8:27b-mlx | Local | 85.0 ±16.3 | 20 | 1 | — | 100.0% | 24.2 s | — |
| claude-sonnet-5-5 | API | 84.0 ±7.3 | 100 | 1 | — | 100.0% | 2.6 s | $95.3 |
| claude-haiku-5-5 | API | 78.0 ±8.1 | 100 | 1 | — | 99.0% | 1.7 s | $18.4 |
| gemma4:12b-mlx | Local | 65.0 ±19.4 | 20 | 1 | — | 100.0% | 11.4 s | — |
| ministral-3:14b | Local | 55.0 ±19.9 | 20 | 1 | — | 100.0% | 53.6 s | — |
| pplx-decider:27b | Decision | 50.0 ±20.0 | 20 | 1 | — | 55.0% | 13.2 s | — |
| d3:27b | Decision | 5.0 ±13.2 | 20 | 1 | — | 10.0% | 29.5 s | — |
| gpt-6.1-sol · thinking on | API | 98.0 ±3.7 | 100 | 1 | — | 100.0% | 7.3 s | $67.6 |
| gpt-5.4 · thinking on | API | 97.0 ±4.1 | 100 | 1 | — | 100.0% | 10.4 s | $94.3 |
| gpt-6.1-terra · thinking on | API | 97.0 ±4.1 | 100 | 1 | — | 100.0% | 4.6 s | $33.2 |
| gpt-6-luna · thinking on | API | 96.0 ±4.5 | 100 | 1 | — | 100.0% | 3.7 s | $3.4 |
| claude-opus-5-5 · thinking on | API | 95.0 ±4.8 | 100 | 1 | — | 100.0% | 3.7 s | $193.7 |
| gemini-4-argon · thinking on | API | 95.0 ±4.8 | 100 | 1 | — | 100.0% | 19.9 s | $81.7 |
| claude-sonnet-5-5 · thinking on | API | 89.0 ±6.4 | 100 | 1 | — | 100.0% | 2.6 s | $96.1 |
| gemini-3.8-flash · thinking on | API | 87.0 ±6.8 | 100 | 1 | — | 100.0% | 10.3 s | $31.2 |
| claude-haiku-5-5 · thinking on | API | 82.0 ±7.6 | 100 | 1 | — | 100.0% | 3.1 s | $18.7 |
| route.question · accuracy · n = 100 | ||||||||
| gpt-6.1-sol · minimum thinking | API | 98.0 ±3.7 | 100 | 3 | 100.0% | 100.0% | 1.8 s | $1.7 |
| pplx-decider:27b | Decision | 95.0 ±4.8 | 100 | 1 | — | 100.0% | 2.7 s | — |
| gpt-6-luna | API | 95.0 ±4.8 | 100 | 3 | 98.0% | 100.0% | 1.1 s | $0.090 |
| gpt-6.1-terra | API | 95.0 ±4.8 | 100 | 3 | 96.0% | 100.0% | 1.3 s | $0.77 |
| d3:27b | Decision | 94.0 ±5.1 | 100 | 1 | — | 100.0% | 3.0 s | — |
| bonsai-2:27b | Local | 94.0 ±5.1 | 100 | 1 | — | 100.0% | 4.9 s | — |
| qwen3.8:27b-mlx | Local | 94.0 ±5.1 | 100 | 1 | — | 100.0% | 3.1 s | — |
| claude-opus-5-5 · minimum thinking | API | 93.3 ±5.3 | 100 | 3 | 99.0% | 100.0% | 1.7 s | $6.4 |
| gemini-3.8-flash | API | 92.3 ±5.6 | 100 | 3 | 95.0% | 100.0% | 2.1 s | $1.2 |
| gpt-5.4 | API | 92.3 ±5.6 | 100 | 3 | 98.0% | 100.0% | 1.5 s | $4.6 |
| gemini-4-argon · minimum thinking | API | 92.0 ±5.7 | 100 | 3 | 98.0% | 100.0% | 6.4 s | $4.0 |
| gemma4:12b-mlx | Local | 92.0 ±5.7 | 100 | 1 | — | 100.0% | 1.3 s | — |
| gemma4:31b-mlx | Local | 91.0 ±5.9 | 100 | 1 | — | 100.0% | 2.5 s | — |
| claude-sonnet-5-5 | API | 90.3 ±6.1 | 100 | 3 | 96.0% | 100.0% | 1.9 s | $3.3 |
| claude-haiku-5-5 | API | 88.7 ±6.4 | 100 | 3 | 97.0% | 100.0% | 1.0 s | $0.17 |
| ministral-3:14b | Local | 81.0 ±7.7 | 100 | 1 | — | 100.0% | 3.1 s | — |
| gpt-6.1-sol · thinking on | API | 97.3 ±4.0 | 100 | 3 | 99.0% | 100.0% | 2.1 s | $2.0 |
| gpt-6.1-terra · thinking on | API | 94.7 ±4.9 | 100 | 3 | 95.0% | 100.0% | 1.6 s | $0.99 |
| claude-opus-5-5 · thinking on | API | 93.3 ±5.3 | 100 | 3 | 98.0% | 100.0% | 1.8 s | $7.3 |
| gemini-4-argon · thinking on | API | 93.3 ±5.3 | 100 | 3 | 98.0% | 100.0% | 9.0 s | $7.9 |
| gpt-5.4 · thinking on | API | 93.3 ±5.3 | 100 | 3 | 96.0% | 100.0% | 3.0 s | $7.2 |
| gpt-6-luna · thinking on | API | 93.3 ±5.3 | 100 | 3 | 94.0% | 100.0% | 1.7 s | $0.13 |
| claude-sonnet-5-5 · thinking on | API | 92.3 ±5.6 | 100 | 3 | 94.0% | 100.0% | 1.7 s | $4.0 |
| claude-haiku-5-5 · thinking on | API | 91.7 ±5.7 | 100 | 3 | 91.0% | 100.0% | 1.1 s | $0.21 |
| gemini-3.8-flash · thinking on | API | 91.0 ±5.9 | 100 | 3 | 98.0% | 100.0% | 5.0 s | $3.9 |
| select.sections · mean_overlap · n = 100 | ||||||||
| gpt-6.1-sol · minimum thinking | API | 93.8 ±5.2 | 100 | 3 | — | 100.0% | 1.4 s | $2.0 |
| gemini-4-argon · minimum thinking | API | 83.8 ±7.3 | 100 | 3 | — | 100.0% | 8.0 s | $5.1 |
| gpt-6.1-terra | API | 82.7 ±7.5 | 100 | 3 | — | 100.0% | 1.1 s | $0.94 |
| gemini-3.8-flash | API | 81.5 ±7.6 | 100 | 3 | — | 100.0% | 2.1 s | $1.7 |
| claude-opus-5-5 · minimum thinking | API | 80.5 ±7.8 | 100 | 3 | — | 100.0% | 1.6 s | $10.5 |
| claude-sonnet-5-5 | API | 78.7 ±8.0 | 100 | 3 | — | 100.0% | 1.6 s | $5.3 |
| gpt-6-luna | API | 73.0 ±8.6 | 100 | 3 | — | 100.0% | 835 ms | $0.10 |
| gemma4:31b-mlx | Local | 72.0 ±8.7 | 100 | 1 | — | 100.0% | 1.7 s | — |
| gpt-5.4 | API | 69.8 ±8.9 | 100 | 3 | — | 100.0% | 849 ms | $1.7 |
| claude-haiku-5-5 | API | 68.0 ±9.0 | 100 | 3 | — | 100.0% | 913 ms | $0.27 |
| gemma4:12b-mlx | Local | 67.5 ±9.0 | 100 | 1 | — | 100.0% | 727 ms | — |
| qwen3.8:27b-mlx | Local | 63.5 ±9.3 | 100 | 1 | — | 100.0% | 1.5 s | — |
| bonsai-2:27b | Local | 55.0 ±9.6 | 100 | 1 | — | 100.0% | 3.7 s | — |
| ministral-3:14b | Local | 46.0 ±9.6 | 100 | 1 | — | 100.0% | 1.7 s | — |
| gpt-6.1-sol · thinking on | API | 93.3 ±13.9 | 20 | 3 | — | 100.0% | 2.9 s | $2.9 |
| gpt-5.4 · thinking on | API | 87.5 ±15.6 | 20 | 3 | — | 100.0% | 6.5 s | $16.3 |
| gpt-6.1-terra · thinking on | API | 85.8 ±16.1 | 20 | 3 | — | 100.0% | 2.1 s | $1.4 |
| gpt-6-luna · thinking on | API | 84.2 ±16.4 | 20 | 3 | — | 100.0% | 2.9 s | $0.24 |
| gemini-3.8-flash · thinking on | API | 80.8 ±17.2 | 20 | 3 | — | 100.0% | 24.3 s | $18.7 |
| claude-sonnet-5-5 · thinking on | API | 80.0 ±17.3 | 20 | 3 | — | 100.0% | 1.1 s | $5.6 |
| gemini-4-argon · thinking on | API | 80.0 ±17.3 | 20 | 3 | — | 100.0% | 15.9 s | $12.4 |
| claude-opus-5-5 · thinking on | API | 76.7 ±17.9 | 20 | 3 | — | 100.0% | 1.5 s | $11.5 |
| claude-haiku-5-5 · thinking on | API | 75.8 ±18.1 | 20 | 3 | — | 100.0% | 1.8 s | $0.41 |
Rows with a different n (for example, a smaller thinking-on sample) are listed but never ranked against each other. “Minimum thinking” marks models whose thinking cannot be turned fully off: their “off” rows ran at the lowest setting.
Models in this set
| Model | Type | Provider | Size · quantisation | Elsewhere |
|---|---|---|---|---|
| claude-haiku-5-5 | API | anthropic | — | Artificial Analysis ↗ |
| claude-opus-5-5 | API | anthropic | — | Artificial Analysis ↗ |
| claude-sonnet-5-5 | API | anthropic | — | Artificial Analysis ↗ |
| d3:27b | Decision | decision | — | — |
| pplx-decider:27b | Decision | decision | — | — |
| gemini-3.8-flash | API | — | Artificial Analysis ↗ | |
| gemini-4-argon | API | — | Artificial Analysis ↗ | |
| gemini-embedding-001 | API | — | — | |
| gemini-embedding-2 | API | — | — | |
| bge-m3 | Local | ollama | — | — |
| bonsai-2:27b | Local | ollama | 26.9B · Q2_0 | — |
| embeddinggemma-2:570m | Local | ollama | — | — |
| embeddinggemma-2:740m | Local | ollama | — | — |
| gemma4:12b-mlx | Local | ollama | 12.4B · nvfp4 | — |
| gemma4:31b-mlx | Local | ollama | 31.7B · nvfp4 | — |
| granite-embedding:278m | Local | ollama | — | — |
| ministral-3:14b | Local | ollama | 13.9B · Q4_K_M | — |
| nomic-embed-text-v2-moe | Local | ollama | — | — |
| paraphrase-multilingual | Local | ollama | — | — |
| qwen3-embedding:0.6b | Local | ollama | — | — |
| qwen3-embedding:8b | Local | ollama | — | — |
| qwen3.8:27b-mlx | Local | ollama | 27.8B · nvfp4 | — |
| snowflake-arctic-embed2 | Local | ollama | — | — |
| gpt-5.4 | API | openai | — | Artificial Analysis ↗ |
| gpt-6-luna | API | openai | — | Artificial Analysis ↗ |
| gpt-6.1-sol | API | openai | — | Artificial Analysis ↗ |
| gpt-6.1-terra | API | openai | — | Artificial Analysis ↗ |
| text-embedding-3-large | API | openai | — | — |
| text-embedding-3-small | API | openai | — | — |
Links go to the model's page on Artificial Analysis, an independent site that benchmarks models at full precision through their APIs, with its own tasks and settings. Their results are not part of GAUGE and are not shown here.