GAUGE · Methodology
How a score is made.
How tasks are sourced, sampled, scored and compared.
How a GAUGE score is made
From the calls a production agent really makes to a published reading, in six steps. Only the last step ever leaves the lab.
Animated diagram of the GAUGE method in six stations: source real agent calls; anonymise them (client names become fictional, secrets are masked); freeze a hashed, versioned case set; run every model with the same settings; score with 95% margins and call ties; publish one reading per model on a dial.
01Scope
GAUGE measures the single model calls that agents make inside production workflows. It does not measure multi-step, tool-using agent runs.
02Task sourcing
Tasks come from production FlowX agents. Inputs are anonymised: client names are replaced with fictional names and credentials are masked. Task ids and descriptions are generic and never name a client or project.
03Case sets
Each case set is frozen and versioned: the sample (seed and n) and a hash of every task file are pinned. A version is never overwritten, and results are compared only within one version.
04Reference answers
Today scores measure agreement with the outputs of the models running in production. Human-checked gold answers replace them before results publish, so no model grades its own outputs.
05Scoring and margins
Each task has one metric: accuracy, mean overlap (section selection) or recall@5 (embeddings). Margins are 95% Agresti–Coull intervals over distinct cases: 1.96·√(p̃(1−p̃)/(n+4)), with p̃ = (pn+2)/(n+4). Unlike the plain Wald form, they do not collapse to zero near a perfect score, so a 100/100 result is not read as a separated winner on its own. A model is tied with the best when its score is at least the best score minus the best score’s margin. Only rows with the task’s full n are compared.
06Settings
Identical prompts, output budgets and settings for every vendor. Thinking is run off and on where a model supports it and reported separately. The served model version is recorded.
07Repeats and consistency
Each case runs several times. Consistency is the share of cases where the repeats agree.
08No fallbacks
A refusal, a timeout or an invalid output counts as a miss. Validity is the share of outputs that parse against the task schema.
09Latency and cost
Local latency is wall time on one Apple M5 Max. API latency is the request round trip, including network and queue. Cost is list price per 1,000 calls at the measured token counts (USD per 1M tokens), without prompt caching.
10Neutrality
No vendor sponsors or reviews results before publication. Each major model release is run within days.