GAUGE · Sovereign AI
Make your AI sovereign.
Pick a task. See the self-hosted model that ties with the best, the hardware it needs, and what it costs each month against the API.
01 Task
Classify an internal operations email into one of 18 business categories, with retrieved category examples and similar earlier tasks as context.
04 Monthly cost
11 models tied at the top on classify.email, not separated at n = 100. 18 rows tied.
- pplx-decider:27b86.0 · tied
- qwen3.8:27b-mlx85.0 · tied
- gemma4:31b-mlx84.0 · tied
qwen3.8:27b-mlxEstimated speed
list price at measured tokens
The API costs 66% less at 300k calls a month.
How the self-hosted figure is made
- 1 × €6,225 list price = €6,225, written off: €173 a month
- Power: 137 kWh a month (0.24 kW max × 0.6 average × PUE 1.3) = €24
- 1 model copy (1 per machine), each serving 4 requests at once.
- Memory per copy: 28.0 GB, the resident memory measured on the bench machine, valid up to its longest prompt (13,923 tokens with output). Assumed, not measured: one cache block per concurrent request.
- Serving 4 requests at once is an assumption: GAUGE measured one request at a time.
| Hardware | Where | Model | Machines | Latency | Per month |
|---|---|---|---|---|---|
| NVIDIA DGX Spark (GB10), 128 GBNVIDIA GB10 | On-prem · NVIDIA | qwen3.8:27b-mlx | 11 copy · 4 at once | 15 sestimate | €197 |
| Mac Studio, M5 Max (18-core CPU, 40-core GPU), 128 GB, 1 TBApple M5 Max (40-core GPU) | On-prem · Apple | qwen3.8:27b-mlx | 22 copies · 4 at once | 6.7 smeasured | €312 |
| Mac Studio, M5 Ultra (36-core CPU, 80-core GPU), 512 GBApple M5 Ultra (80-core GPU) | On-prem · Apple | qwen3.8:27b-mlx | 11 copy · 4 at once | 4.8 sestimate | €593 |
| g6e.xlargeNVIDIA L40S | AWS | qwen3.8:27b-mlx | 11 copy · 2 at once | 4.8 sestimate | €959 |
| Standard_NC72lds_xl_RTXPRO6000BSE_v6NVIDIA RTX PRO 6000 Blackwell Server Edition | Azure | qwen3.8:27b-mlx | 11 copy · 2 at once | 5.2 sestimate | €1,327 |
| Server with 2 × RTX PRO 6000 Blackwell Server Edition (96 GB each)NVIDIA RTX PRO 6000 Blackwell Server Edition | On-prem · NVIDIA | qwen3.8:27b-mlx | 11 copy · 4 at once | 2.6 sestimate | €1,458 |
| g2-standard-48NVIDIA L4 · 2 GPUs per copy | Google Cloud | qwen3.8:27b-mlx | 11 copy · 2 at once | 8.1 sestimate | €1,943 |
| Standard_NC24ads_A100_v4NVIDIA A100 80GB PCIe | Azure | qwen3.8:27b-mlx | 11 copy · 4 at once | 2.1 sestimate | €2,041 |
What is measured
GAUGE runs every local model on one Apple M5 Max (40-core GPU) (128 GB, 614 GB/s), Ollama 0.40.15. For each task it records decode and prefill speed, wall-time latency, token counts and the memory the loaded model occupied. On that machine the calculator uses these numbers as they are.
The models the export lists as tied with the best on the task are the self-hosted options. Where no local model ties, you can still price one that scored lower; the calculator then shows its score gap and marks every figure as not equivalent. Scores and ties come from the export; nothing is re-ranked or averaged across tasks.
What is estimated
Other hardware is scaled from the measurement. Decode is limited by memory bandwidth, so it scales with the device's bandwidth. Prefill is limited by compute: 35% of the device's dense FP16 peak, divided by two operations per active parameter. A model split over 2, 4 or 8 GPUs keeps 85%, 75% or 70% of their combined bandwidth. Serving stacks built for NVIDIA GPUs are often faster than this; treat estimates as a cautious starting point, not a quote.
Memory comes from the measurement. Where the runtime reports the context it loaded (llama.cpp), memory per request is the measured cache divided by that context, sized for the task's longest prompt plus its 95th-percentile output. Where it does not (MLX), the measured resident memory is used as an envelope that holds up to the longest prompt measured, and each extra request served at once is charged the whole measured cache again. GAUGE runs every local model one request at a time, so any setting above one request at once is an assumption, not a measurement. Prompt speed is taken from the largest prompt, where the prompt cache helps least.
Load and cost
Capacity is sized for a busiest hour 1.5× the average, at 70% utilisation. Business hours are 10 hours on 22 days (220 h a month); on-demand cloud machines are stopped outside them, committed ones bill all 730 h.
On-prem is the purchase price written off over the chosen period plus power: 60% of the box's maximum draw on average, times a PUE of 1.3, at the country's non-household electricity price. Staff, rack space, network and support are not included, for cloud or on-prem.
API cost is the export's list price per 1,000 calls at the measured token counts, without prompt caching.
Sources
- AWS: Price List API, fetched 2026-11-09
- Azure: Retail Prices API, fetched 2026-11-09
- Google Cloud: Cloud Billing Catalog API, fetched 2026-11-09
- EUR/USD: ECB reference rate 1.1165 (2026-11-06)
- Electricity: Eurostat nrg_pc_205, 2,000–19,999 MWh/year, excl. VAT, 2026-S1
- On-prem prices: vendor and reseller list prices, checked by hand; servers are indicative
- Device specs: vendor datasheets, checked 2026-11-08
Cloud prices refresh every day.