Day 7 · Format drift · collected 2026-09-14

Fifteen models flipped from JSON to prose in seventy-two hours

Fifteen models ran the same red-team block under an unchanged prompt. Not one returned the required schema. Every model invented its own verdict vocabulary, and most of them answered in prose.

31
entries
15
models
0 / 15
schema compliance
14
prose responses
Presenter mode shows one section per screen. Arrow keys move, A toggles autoplay, Esc exits.
1
Ranking

How the field ranked on this day

How far each model got, of 3 steps
Nobody finished the task. Step 1 is a file that parses as JSON, step 2 is a usable verdict key inside it, step 3 is the full seven-key schema. Eight models reached step 2, three parsed but named no verdict, and four never produced parseable JSON.
deepseek2 / 3gemini2 / 3glm flash*2 / 3grok2 / 3kimi2 / 3longcat2 / 3sonnet2 / 3yuanbao2 / 3mimo1 / 3minimax1 / 3qwen1 / 3chatgpt0 / 3meta0 / 3mistral0 / 3solar0 / 3
top value groupbottom value groupbetween
No model reached step 3: the best result on any single key was 1 of 7, and the verdict vocabulary was invented afresh by each one. chatgpt (12.1 KB), meta (6.0 KB), mistral (two files) and solar (5.4 KB) returned files that do not parse at all, which is the hollow marker. mimo, minimax and qwen parsed cleanly but carried no verdict key. * The free web chat build of GLM 5.3 Flash is the quantized one; the API-harness deployment of the same model name is a different build, and the two do not score alike.
2
The prompt

A literal copy-exact template and a first-character contract

JI-v1.1 was already the hardened version: it added a binding output-format contract, because an earlier block-01 wave had returned prose reviews from most models despite a “nothing else” rule. The schema was still described abstractly. That turned out to be the problem.

Goal of this animation

Show Day 7's format drift: a schema described in one line of prose, and what fifteen models returned instead.

the binding contractfirst character {last character }seven keys described in one lineverdict from a four-value enumSOUND | SOUND-WITH-RESERVATIONSSUSPECT | IMPLAUSIBLE-FOR-THIS-RECORDchatgptdeepseekgeminiglm flashgrokkimilongcatmetamimominimaxmistralqwensolarsonnetyuanbaorequired keys present, out of sevenverdicts the models inventedrejectreject_and_revisereject-as-isoverstatedREVISE_BEFORE_INGESTnot_ready0 / 15 carried the schema
Step 4 of 4

Zero of fifteen

None of the fifteen carried the schema. Day 8 asked the same thing a different way.

Left and right arrow keys step back and forward once the player has focus. Link to a step with #xp-day7-step=N.
J-ISSUE-REDTEAM-V1 · JI-v1.1 (block 01)superseded on 2026-09-10 by JI-v1.2, which replaced the abstract schema with a literal template
**OUTPUT FORMAT (binding, read first):** your entire response must be exactly one JSON object matching the schema at the end of this brief. The first character of your reply must be `{` and the last must be `}`. No markdown, no headings, no commentary before or after, no code fences, no explanations — the JSON object only.

You are a skeptical principal engineer reviewing a DRAFT incident episode before it enters a curated dataset. The episode was mined from a public bug-tracker thread. Your job is to attack it: find every technical claim that is wrong, implausible, or internally inconsistent — and say what a correct version would look like.

**The draft under review:** RECORD: OpenROAD issue #6066 — BoomTile/Megaboom: 20k routing congestion DRCs at only 0.24 placement density.

**Your review, as a single bare JSON object:** {"verdict": one of SOUND | SOUND-WITH-RESERVATIONS | SUSPECT | IMPLAUSIBLE-FOR-THIS-RECORD, "objections": [...], "mechanism_check", "evidence_check", "attribution_check", "missing_from_story", "most_defensible_part"}

**Hard rules:** attack the NUMBERS, the causal chain, and whether the intervention explains the outcome — not the writing. Do not browse the web or fetch any page. Fewer sharp objections beat a padded list.
The drift is the finding: 0 of 15 files carried the required seven-key schema, and every model invented its own verdict vocabulary: reject, reject_and_revise, REVISE_BEFORE_INGEST, overstated, not_ready, reject-as-is.
3
What graded it

The Flash station and a prompt fix

Station
GLM 5.3 Flash
Collected and harvested the block-01 wave. The result was a prompt-format problem rather than a prompt-format problem, and it was recorded as one.
Web Model Tasks/Day 7/v12-pilot/
The fix
JI-v1.2: a literal copy-exact template
The abstract one-line schema was replaced by a literal template with field-leading enums (HOLDS/BREAKS/N-D, MEAS/ASSERT/N-D, SUPPORTS/WEAK/ABSENT/N-D), after this pilot returned 0/15 schema compliance and every model invented its own verdict vocabulary.
prompts/J-ISSUE-REDTEAM-V1.md (version note)
Note
A 72-hour flip on an unchanged prompt
The same family of prompts had returned compliant JSON earlier in the week. The drift is a property of the chat surface rather than of the schema.
digest/DIGEST-REPORT.md
4
How each model performed

Response form and schema compliance measured from the files

Measured from Web Model Tasks/Day 7/: every file inspected and parsed
ModelJSON fileProse file7-key schemaVerdict token the model actually emitted
chatgptyes (12.1 KB, unparseable)yes0 / 7n/a
deepseekyes (4.9 KB, parses)yes1 / 7reject
geminiyes (5.1 KB, parses)yes1 / 7reject_and_revise
glm flashyes (15.1 KB, parses)yes1 / 7reject-as-is
grokyes (3.7 KB, parses)yes1 / 7overstated
kimiyes (6.6 KB, parses)yes1 / 7REVISE_BEFORE_INGEST
longcatyes (7.2 KB, parses)yes1 / 7REJECT for curated dataset. The record contains fundamental stage-confusion errors…
metayes (6.0 KB, unparseable)yes0 / 7n/a
mimoyes (9.3 KB, parses)yes0 / 7n/a (fenced, no verdict key)
minimaxyes (6.7 KB, parses)yes0 / 7n/a (no verdict key)
mistralyes (2 of them, neither parses)yes0 / 7n/a
qwenyes (2.4 KB, parses)yes0 / 7n/a (no verdict key)
solaryes (5.4 KB, unparseable)yes0 / 7n/a
sonnetyes (4.3 KB, parses)yes1 / 7not_ready
yuanbaoyes (13.9 KB, parses)yes1 / 7reject-as-recovery-episode
Every file was opened and measured: 16 JSON files and 14 prose files. No file carried the required seven-key schema. The best any model managed was one key out of seven. Four files do not parse at all. The verdict column shows what the models emitted instead: fifteen different vocabularies for a four-value enum.
5
What it found

The finding is the drift itself

Best performer
longcat
One of eight models at the day’s ceiling of one schema key, and its verdict named the record’s stage-confusion errors.
Weakest performer
mistral
Returned two JSON files and neither parsed, the only model to fail twice, with zero of seven schema keys.
Zero of fifteen returned the required schema
The prompt offered four verdict values. The fifteen models emitted reject, reject_and_revise, REVISE_BEFORE_INGEST, overstated, not_ready, reject-as-is, reject-as-recovery-episode and more: fifteen vocabularies for one enum. A grader keyed on the enum would have failed all fifteen for the wrong reason, and shortest-enum-first matching would have collapsed reserved verdicts into SOUND, the defect found in the Day-5 codebook.
The contract broke on the prose files
The rule asked for one bare JSON object, no fences and no commentary. Fourteen of the fifteen models also returned a prose file, and mistral returned two JSON files with no prose at all. Two of the sixteen JSON files open with a ```json fence the prompt forbids by name, mimo’s and mistral’s second. The other fourteen open with {.
The fix was structural and it worked
JI-v1.2 replaced the one-line schema with a literal object to copy and put the enum values directly against the field names. The same three models scored 0 of 15 on the one-line schema and 30 of 30 on the template, all seven keys present, none renamed or extra.
The drift was a delivery problem
kimi’s prose response names a confounded intervention, two knobs changed at once, which is a real objection the schema had no field for. deepseek wrote “reject” where the enum held four reserved values and none of them was “reject”. Both models reasoned correctly and encoded incorrectly.
6
Source files

Where the evidence lives

🔒 All inputs are processed locally in your browser. No data is transmitted. Do not enter real personal data — use synthetic or anonymised inputs only.