Day 8 · The sweep · collected 2026-09-17

Thirty schema-compliant reviews of ten incident drafts

The JI-v1.2 fix was tested against the same three models that had just produced 0 / 15 schema compliance. This time every one of the thirty responses parsed on the first try, with the full seven-key schema and no renamed keys.

30
files
3
models
10
drafts
30 / 30
schema compliance
15 / 15
SUSPECT / S-W-RES
Presenter mode shows one section per screen. Arrow keys move, A toggles autoplay, Esc exits.
1
Ranking

How the field ranked on this day

Blocks called SUSPECT, of 10
Higher means stricter. All three models passed the seven-key schema on all thirty responses, so this chart is not an accuracy ranking: it shows how often each called a draft SUSPECT rather than SOUND-WITH-RESERVATIONS.
deepseek9/10sonnet4/10kimi2/10
a strictness rank rather than an accuracy rank
chatgpt is absent: it ran a single pilot block. On this day the three models tied on the graded metric (30/30 schema), which is why the index chart ranks them level and this chart shows strictness instead.
2
The prompt

The literal template that replaced the abstract schema

JI-v1.2 is the whole intervention. The Day-7 pilot had shown that an abstract one-line schema was not a contract: fifteen models each invented their own verdict vocabulary. JI-v1.2 replaced the schema with a literal object to copy, and put the enum values directly against the field names.

Each model saw each of the ten drafts exactly once. The drafts were stratified by failure type rather than taken as the first ten: unmeasured post-state, resolution claimed with no fix described, a tooling policy that hides errors, a crash on degenerate input, measurement validity, nondeterminism, silent netlist corruption, a repair that took 40000 seconds on one endpoint, a misattributed root cause, and a bug closed without reproduction.

Goal of this animation

Show the Day 8 sweep: the schema given as a literal object to copy, and three models parsing cleanly on all thirty responses.

a literal object to copy{"verdict": "SOUND | SUSPECT | …","objections": [ … ],"mechanism_check": "…","evidence_check": "…","attribution_check": "…","missing_from_story": "…","most_defensible_part": "…"}the ten incident drafts12345678910kimisonnetdeepseek30 / 30 parsed on the first tryall seven keys present and no key renamed
Step 5 of 5

What changed from Day 7

Day 7's prose schema got zero of fifteen. Giving the object to copy is what changed.

Left and right arrow keys step back and forward once the player has focus. Link to a step with #xp-day8-step=N.
J-ISSUE-REDTEAM-V1 · JI-v1.2the fix that closed the Day-7 drift: a literal template instead of a described schema
**OUTPUT FORMAT (binding, read first):** your entire response must be exactly one JSON object matching the schema at the end of this brief. The first character of your reply must be `{` and the last must be `}`. No markdown, no headings, no commentary before or after, no code fences, no explanations — the JSON object only.

**Your review, as a single bare JSON object:** {"verdict": one of SOUND | SOUND-WITH-RESERVATIONS | SUSPECT | IMPLAUSIBLE-FOR-THIS-RECORD, "objections": [{"claim_attacked", "objection", "severity", "confidence"}], "mechanism_check", "evidence_check", "attribution_check", "missing_from_story", "most_defensible_part"}

**Hard rules:** attack the NUMBERS, the causal chain, and whether the intervention explains the outcome — not the writing. The pasted record is the WHOLE evidence base: do not browse the web or fetch any page. Do not invent thread content, commits, PRs, participants, or tool behaviour that is not in it. If the record does not settle a question, write “not-determinable-from-this-record” rather than guessing — a field-level escape beats a guessed objection. A claimed resolution with no measured post-state is an EVIDENCE defect, not a mechanism defect; classify it under `evidence_check`. If a claim is sound, say so — inventing objections is as bad as missing real ones. Fewer sharp objections beat a padded list. This review is a PRE-SCREEN for expert review, not a verdict on the underlying bug — flag, don’t convict.
Same three models as the Day-7 pilot: kimi, sonnet, deepseek. Chosen for clean records on the earlier truth-anchored instruments: arithmetic exact on 9 / 9 values, format conformance 4 / 4 on the reference canonicalizer, correct on the citation-anchor and measured browser-behaviour probes. chatgpt ran block 01 as a format pilot and was schema-compliant, but was not extended to the other nine blocks.
3
What graded it

How the sweep was harvested and what debt it left

Station
GLM 5.3 Flash
Collected and harvested the sweep. The harvest is reproducible from the script rather than from a human reading thirty files: digest/_verify/flash_d6_harvest.py → d6run/d6harvest.json, staged tally in redteam/TALLY-34.md.
digest/wave1-report.md
The instrument
Format conformance, then verdict calibration
Two separate checks. First the mechanical one: does the file parse, does it carry all seven keys, are any keys renamed or extra. Then the calibration check: the prompt defines a four-value enum with two reserved values, so a bare SOUND on a draft known to be imperfect would be the same miscalibration defect the tally codebook had produced earlier.
redteam/TALLY-34.md
Recorded debt
Expert review: not performed
None of the thirty verdicts has been reviewed by a human expert. The dataset card, when it ships, will say exactly that. The verdicts and the full response JSON are on disk and hash-verified, so anyone can re-run the same prompt on the same block and compare.
digest/wave1-report.md, “What the review debt looks like”
4
How each model performed

Schema compliance and verdict calibration measured from the files

Measured from Web Model Tasks/Day 8/: all 30 files parsed without repair
ModelFiles7-key schemaObjectionsMajor / minorSUSPECTS-W-RESBare SOUND
deepseek10 / 1030 / 302620 / 6910
kimi10 / 1030 / 302316 / 7280
sonnet10 / 1030 / 30228 / 14460
chatgpt1 (pilot)block 01 onlyn/an/an/an/a0
Per-block verdict matrix; S-W-RES = SOUND-WITH-RESERVATIONS
BlockDraft under reviewdeepseekkimisonnetReading
red1-01BoomTile/Megaboom: 20k congestion DRCs at 0.24 densitySUSPECTS-W-RESS-W-RESdeepseek strictest
red1-02GPL-0314: getFillerGCell index out of boundsSUSPECTSUSPECTSUSPECTunanimous
red1-03RUDY and global-route congestion maps disagree (ICCAD04 risc2)SUSPECTSUSPECTS-W-RESstrong concern
red1-04GRT DRC cap (10000) hides hotspots and macro overlapsSUSPECTS-W-RESSUSPECTsplit
red1-05repair_timing ODB-1200 with Liberty Q→QN arcsSUSPECTS-W-RESS-W-RES
red1-06TritonCTS segfault when Liberty has no max_capacitanceSUSPECTS-W-RESSUSPECTsplit
red1-07CTS NDRs on heavily gated clocks (level definition)SUSPECTS-W-RESSUSPECTsplit
red1-08asap7 cva6 hier-synth GRT report references a nonexistent netSUSPECTS-W-RESS-W-RES
red1-09STA false hold/setup: ideal top clock mixed with propagated macro clockS-W-RESS-W-RESS-W-RESunanimous; strongest record
red1-10GPL regression: too-early non-virtual TD iteration makes design unroutableSUSPECTS-W-RESS-W-RESdeepseek strictest
Thirty files, thirty first-try parses, thirty complete seven-key objects, zero renamed or extra keys. Verdict distribution: 15 SUSPECT, 15 SOUND-WITH-RESERVATIONS. No SOUND, no IMPLAUSIBLE. Zero fatal objections across all thirty reviews. deepseek flagged SUSPECT on nine of ten blocks; kimi and sonnet leaned SOUND-WITH-RESERVATIONS. The models agree on which records are strong and which are thin.
5
What it found

The contract held and the disagreement was the finding

Best performer
deepseek
Produced the most objections, 26 with 20 major, and was the strictest model on two of the ten blocks.
Weakest performer
sonnet
The least strict model: 8 major objections out of 22, and 4 blocks flagged suspect where deepseek flagged 9.
The literal template fixed the drift
Day 7 produced 0 of 15 schema compliance from a schema described in one line of prose. Day 8 produced 30 of 30 from the same three models against a literal object to copy, with no renamed or extra keys. The prompt family, the models and the chat surface were unchanged.
No model invented content and none refused
Across thirty reviews there are no fabricated quotes, no invented tool or chip names, no invented commits and no invented participants. No model browsed the web and none refused. The field-level escape not-determinable-from-this-record was available, and every model treated the pasted record as its whole evidence base.
Zero fatal objections reflects the models’ caution
The enum offers two reserved values, SUSPECT and IMPLAUSIBLE-FOR-THIS-RECORD. The second was never used by any of the three models on any of the ten blocks. Every draft carries a known defect by construction, since the set was stratified by failure type, and no model escalated past SUSPECT. The enum is wider than the models’ willingness to convict.
Unanimity tracked the record
Block 02 resolves the incident with the sentence “solved on the latest version”, no commit and no PR, and drew SUSPECT from all three models. Block 09, the STA false hold and setup record, is the best evidenced in the set and drew SOUND-WITH-RESERVATIONS from all three. Where the three agree on which records are thin, a human reviewer can start there.
No model emitted a bare SOUND on block 01
This is the check the Day-5 codebook failed. Shortest-alias-first matching collapses SOUND-WITH-RESERVATIONS into SOUND and manufactures a clean verdict. Across all thirty files no model emitted a bare SOUND, so no grader had to collapse a reserved verdict to reach one. deepseek reached SUSPECT on 9 of the 10 blocks.
Three models from one generation is a narrow basis
deepseek, kimi and sonnet share training biases we cannot see. Thirty reviews of ten drafts describe how those three read incident drafts. Three models from one generation cannot stand for the web field. The drafts are first-generation mining with no expert review yet, and if that review changes them these verdicts describe the old versions.
6
Source files

Where the evidence lives

Web Model Tasks/Day 8/ (30 files, deepseek · kimi · sonnet × red1-01..10)digest/wave1-report.md (the Day-8 report)prompts/J-ISSUE-REDTEAM-V1.md (JI-v1.2)prompts/_paste/redteam/red1-01.paste.md … red1-10.paste.mdprompts/_paste/redteam/MANIFEST-PASTE-RT.sha256digest/_verify/flash_d6_harvest.py → d6run/d6harvest.jsonredteam/TALLY-34.md (staged tally)RUNSHEET-34.md (the 34-draft sheet)
๐Ÿ”’ All inputs are processed locally in your browser. No data is transmitted. Do not enter real personal data โ€” use synthetic or anonymised inputs only.