Between 31 August and 17 September 2026, a structured prompt was pasted by hand into the free web chat interface of each service and the reply saved verbatim. Nothing was automated and nothing was paid for. Every response was then graded, most of it by mechanical recomputation and the paper-recall slices against the papers themselves.
minara, minimax, opus 5, tencent yunbao. * The free web chat build of GLM 5.3 Flash is the quantized one; the API-harness deployment of the same model name is a different build, and the two do not score alike.Each day is a separate page carrying three things: the prompt that was pasted, what graded it, and a table of how every model performed. Where a number could be measured from the response files rather than read out of a summary, it was measured.
The corpus is not expert data and does not pretend to be. No episode invented a chip, company or person. Each verdict is a flag for a human reviewer.
Show how every reply in this benchmark was collected and graded, from one fixed prompt to a verdict that traces back to a measured row.
Every verdict on these pages traces back to a measured row in that day's table, so a reader can check any claim against the primary files.
| Day | Date | Theme | Files | Models | Prompt slices | Graded by |
|---|---|---|---|---|---|---|
| Day 1 | 2026-08-31 | Exploration | 236 | 24 | 19 lettered prompts | Fable 5 |
| Day 2 | 2026-09-01 | Compliance batch | 141 | 15 | 10 regulatory families | Opus 5 |
| Day 3 | 2026-09-03 | Five families | 70 | 15 | boundary · APRC · POWER4 · diagnosis · SDC | Opus 5 |
| Day 4 | 2026-09-04 & 09-08 | Paper recall | 73 | 16 | Alpha · Pentium 4 · POWER4 phys · Niagara · LVS | Opus 5 |
| Day 5 | 2026-09-08 & 09-09 | Five instruments | 80 | 16 | Reg Z · JCS · E · red1 · red3 | Opus 5 + GLM 5.3 Flash |
| Day 6 | 2026-09-11 | Citation + deployment | 32 | 16 | B-v8 · C-v5 | GLM 5.3 Flash |
| Day 7 | 2026-09-14 | Format drift | 31 | 15 | JI-v1.1 block 01 | GLM 5.3 Flash |
| Day 8 | 2026-09-17 | The sweep | 30 | 3 | JI-v1.2, ten blocks | GLM 5.3 Flash |
minara / minaria) and is counted once.sonnetmistraldeepseek and minimax m3 both called the POWER4 coherence protocol directory-based, in a paper where “snoop” appears 39 times and the L3 directory is a cache tag directory. gemini, grok and kimi each placed a single shared FPU across the eight cores of a paper that never uses the word FPU. Ten models agreed the ASIL B SPFM target is 90%, against two public sources that disagree at 90 and 97.yuanbao, solar, minimax 2.7 and mistral all multiply where the IRB grouping requires division, and every one of them closes arithmetically on its own inputs. Nor does it catch a different convention, which is how mistral’s inverted SPFM survived both the S and T slices unchanged.solar pro returned five blocks where the prompt asked for three episodes. yuanbao wrapped its answer in a {"prompt_a": ...} object. minimax m3 lost the "le_amount": key mid-object. solar answered the Reg Z prompt in prose. Each was recorded as an absence or a format note rather than as a wrong answer.Free web chat models are what most people use, and almost nothing published about them measures the product a person gets in a browser tab. Published evaluations run a model through an API, inside a harness, temperature pinned. Every number below comes from a browser tab.
So we chose tasks whose answers can be checked without trusting the model. Financial-regulation calculations were recomputed from the primary text. Recall slices were scored against the papers themselves. Incident reviews were judged against a record that was its own evidence base.
The financial family is the control. It is the same kind of task as the semiconductor calculations in this corpus, a deterministic rule behind a citable clause. Its figures are published and its answers already checked, so a wrong response can be demonstrated by pointing at the number.
We ask about semiconductors because we hold the primary sources. An EDA incident arrives with its bug-tracker thread; an architecture claim sits in a public paper. A reply can be checked against that record rather than our judgement, and the terminology is narrow enough that a bluff is obvious.
We re-ran the work where the answer allowed it: about 720 calculations on Day 2, and the RFC 8785 vectors diffed against a reference canonicalizer. A wrong number stays wrong however the response is worded, which is not true of a score awarded by another model.
We also wanted to keep apart two failures usually blended into one score. A response that ignores a format contract has failed at delivery, and its reasoning may have been sound. A delivery failure is repaired with a template. A reasoning failure needs a different model. Day 7 and Day 8 are the same models on the same prompt family with opposite results, which is only visible if you keep them apart.
Nothing here needed a budget. Every prompt is on disk and every reply was saved verbatim, so anyone with a browser can re-run a block and compare. The method does not ask the reader to trust us.
The thirty red-team verdicts have not been reviewed by a physical-design expert yet. The sweep also ran on three models from one generation, so it describes those three models on ten drafts. Nothing wider follows.