Research · Notes
Notes from the lab.
Short, dated findings: what we tried, what the numbers said, and the data behind it.
- 2026-11-09
For retrieval, local embedding models tie the paid APIs.
On routing-example retrieval (n = 201), snowflake-arctic-embed2, run locally, has the top point estimate at 38.2 recall@5. Five other models sit inside its ±6.7 margin, among them both OpenAI embedding models. The two Gemini embedding models fall below it.
On retrieving the labelled examples the classifier is shown (n = 100), all 13 embedding models tie, local and API alike, within ±8.4 of the top estimate (24.8, text-embedding-3-large). On both tasks, a model you run yourself scores within the margin of the best paid one.
Recall here means overlap with the top 5 the production platform retrieves, not with a human-judged answer set. The absolute numbers are low for that reason. They say how often a model finds what production finds.
Source: GAUGE set v4, tasks embed.example_retrieval and embed.task_retrieval (research.flowx.ai/benchmarks/embeddings)
- 2026-11-09
A self-hosted model ties the best on two of the four language tasks.
On email classification (n = 100), three local models fall inside the ±6.3 margin of the top estimate: pplx-decider:27b (86.0), qwen3.8:27b-mlx (85.0) and gemma4:31b-mlx (84.0). The top estimate is gpt-5.4 at 89.3. gpt-5.4 is also the production model whose answers serve as the reference on this task, so it is scored against its own outputs.
On question routing (n = 100), pplx-decider:27b at 95.0 ties gpt-6.1-sol at 98.0 (±3.7).
On section selection and legal case matching, no local model ties. The local legal results were measured on 20 cases only and sit outside that comparison. The sovereign calculator prices the tied local models against the API, task by task.
Source: GAUGE set v4, tasks classify.email, route.question, select.sections, match.legal_case (research.flowx.ai/benchmarks/sovereign)
- 2026-11-09
Five of six GAUGE tasks have no single winner.
On set v4, only section selection separates one model: gpt-6.1-sol at 93.8 (±5.2). On the other five tasks, between 2 and 13 models sit inside the 95% margin of the top score. GAUGE reports those as ties.
A leaderboard that ranks by point estimate alone would name a winner on all six tasks. On five of them the data does not support one. When several models tie, the choice comes down to price, latency and where the model can run.
Source: GAUGE set v4, winners and margins as exported by the harness (research.flowx.ai/benchmarks)