GAUGE · FlowX Enterprise Agent Task Benchmark
Models, measured on real agent work.
GAUGE reads the single model calls that agents make inside production workflows: routing, classification, section selection, case matching, extraction, grounded answers and retrieval. It does not measure multi-step, tool-using agent runs. Every score links to one frozen case set.
29 models · 6 tasks · set v4
- gpt-5.4 · API89.3
- claude-sonnet-5-5 · API—
- claude-sonnet-5-5 · API—
- claude-opus-5-5 · API—
- claude-opus-5-5 · API—
- gpt-6-luna · API—
Four instruments per model.
Every model is read on the same four dials, on every task, with identical settings across vendors.
Agreement with the reference answer, with the 95% margin as the amber band. Overlapping bands are a tie.
Median time per call: wall time for local models, the full round trip for APIs. Log scale.
List price per 1,000 calls at the measured token counts. Local models have no per-call price.
How often repeated runs of the same case give the same answer.
Four kinds of agent work.
Every dot is one model on one task, placed by its score. Amber marks the best and the models tied with it; filled dots are API models, rings are local, squares are decision models.
Pick one label or handler for an input.
Classification
2 tasks
- classify.email11 models tied at the top, not separated at n = 100
- route.question4 models tied at the top, not separated at n = 100
Decide whether two documents match on a stated criterion.
Cross-checking
1 task
- 2 models tied at the top, not separated at n = 100 · +7 on a different sample
Retrieve the right items by vector similarity.
Embeddings
2 tasks
- embed.example_retrieval6 models tied at the top, not separated at n = 201
- embed.task_retrieval13 models tied at the top, not separated at n = 100
Pick the relevant items from a list.
Selection
1 task
- best gpt-6.1-sol (minimum thinking) · mean_overlap 93.8 · 1 tied · +9 on a different sample
Every task starts as a model call a production FlowX agent makes.
How a task gets into GAUGE
Method
- Inputs anonymised: fictional client names, masked credentials
- Case set frozen and versioned; never overwritten
- Every model scored with the same prompts and settings
Is there a model call your agents make every day that GAUGE should measure?
Read the methodology →Suggest a task →Planned next
Structured extraction
Turn an unstructured message into typed fields.
Grounded QA
Answer only from retrieved documents and say when the context is not enough.
Cross-language answering
Answer in one language from context in another.
29 models on 6 tasks, each scored on one frozen case set (set v4).
How a GAUGE score is made
From the calls a production agent really makes to a published reading, in six steps. Only the last step ever leaves the lab.
Animated diagram of the GAUGE method in six stations: source real agent calls; anonymise them (client names become fictional, secrets are masked); freeze a hashed, versioned case set; run every model with the same settings; score with 95% margins and call ties; publish one reading per model on a dial.
Same tasks, same settings, every model.
Each major model release is run within days. Results are compared only within one frozen set version.