Every model in this benchmark answers the same nine frozen finance tasks three ways: alone, with the AINumbers MCP offered, and with one sentence directing it to the tools. This page walks through the three arms and what the scorer counts, then reports the verified numbers. Results below were recorded 2026-10-15 and are current as of that date.
Each model sees the same task text three times, once per arm. Only what surrounds the prompt changes; the words of the task never do.
The nine tasks are finance computations drawn from the AINumbers catalog: escrow interest, coverage ratios, op-risk capital, a reserve requirement, annuity values and their neighbours. Prompts, scorer and manifests were frozen before the cells ran, and every tool call a model makes is counted mechanically on the harness side.
The bare arm is the baseline. The model receives the task and nothing else, and it must reason to an answer in text.
Clean cells run one fresh session per task in an isolated working directory, so the model cannot reach the benchmark's own files. Where a harness could not isolate the working directory, the run's transcripts were audited and the run carries that disclosure with its numbers.
The AINumbers MCP is mounted and its tool catalogue is visible. Nothing tells the model to use it; usage here measures whether the model goes looking on its own.
Offered-arm usage turned out to be a property of the model, and the measured spread runs from a model that used the tools on every row to models that barely engaged the catalogue at all. That spread is why the benchmark reports usage alongside accuracy instead of assuming the tools were used.
The directed arm keeps the mount from the offered arm and adds a single instruction: find the tool first, then call it. The task text is still untouched.
On the frontier models the directed arm moved usage without moving accuracy, and on one model it added a measurable latency cost for identical answers. The benchmark treats direction as a separate experimental variable for exactly that reason.
Scoring is mechanical and strict: the denominator is all nine tasks, every run. A timeout, a transport error, or an unparseable answer is a miss, because delivering a final answer is part of the task.
The accuracy figures below use this strict full-denominator convention throughout. An earlier relaxed convention that scored only parseable answers was replaced, and every published cell was rescored under the strict rule before the numbers here were signed off.
Nine tasks per arm, nine models with a measured offered arm, recorded 2026-10-15. Cells still pending are marked as such; nothing on this page is projected.
| Model | Bare | MCP (offered) | MCPd (directed) | Note |
|---|---|---|---|---|
| Muse Spark 1.3 | 88.9% | 100% | 100% | 16 recorded tool calls on offered, 29 on directed |
| GLM-5.3-Flash | 88.9% | 100% | pending | 44 ainumbers calls on the offered arm |
| GPT-5.6 Terra | 88.9% | 100% | n/a | 16 ainumbers calls; one sequential session |
| DeepSeek V4.1 (clean) | 77.8% | 100% | 100% | bare is the clean re-run baseline |
| Nemotron 3 Ultra | 77.8% | 100% | 100% | 12 recorded tool calls on offered, 31 on directed |
| Longcat 2.5 | 77.8% | 100% | 100% | 15 recorded tool calls on offered, 30 on directed |
| MiMo V2.6 (clean) | 66.7% | 100% | 100% | largest gain over a clean bare baseline |
| Solar Mini 4 | 88.9% | 77.8% | 88.9% | one wrong value and one transport error on the offered arm |
| Solar Pro 4 (historical) | 66.7% | 88.9% | 88.9% | route died mid-study; a dated re-run on a new route is pending and is not pooled here |
| Sonnet 5 | 88.9% | pending | pending | offered and directed re-runs pending |
| Nemotron 3 Super | 77.8% | not run | not run | bare only: the harness context wall blocked the assisted arms for this model |
Absolute accuracy on a public-task corpus is always contamination-suspect, so the benchmark's claims are about the differences between arms on identical tasks, measured one day at n=9 per arm, and the per-cell disclosures from the run logs travel with any reuse. The full per-task ledger, prompt bytes and scoring manifest live in the project's research notes; a public dataset export is planned and will be published later.