Four papers reconstructed from memory, each graded against a locator-cited extraction set and the full PDF text. This is the only slice in the collection where a model is scored against the source rather than against another model.
glm 5.3 flash resubmitted the same Alpha 21264 text for eight different prompts, which is why 16 of its 20 entries sit outside the paper. * The free web chat build of GLM 5.3 Flash is the quantized one; the API-harness deployment of the same model name is a different build, and the two do not score alike.Each prompt named one paper by its full citation, forbade general knowledge from standing in for paper content, and made a direction error the single worst possible output. Every entry had to carry its own weakest_point.
Show how Day 4 tested paper recall: one named paper, answered from memory, with every claim checked against the paper itself.
Each claim is checked against a locator-cited extraction set and against the full text of the PDF.
This task is about one specific published paper: "The Alpha 21264 Microprocessor" — R. E. Kessler — IEEE Micro, Vol. 19, No. 2, March–April 1999, pp. 24–36.
From your knowledge of THIS SPECIFIC PAPER (not your general knowledge of the Alpha 21264 or DEC), reconstruct up to 5 engineering-decision episodes that the paper itself describes — design choices weighed, alternatives considered or rejected, trade-offs made and their stated reasoning.
...`quote_as_recalled`: the single most load-bearing sentence AS YOU RECALL IT WORD-FOR-WORD (max 25 words; if you cannot reproduce it verbatim, write "cannot-reproduce-verbatim" instead of approximating) ... `not_in_paper`: true ONLY if you believe the entry's claim is NOT actually in this paper ... `paper_vs_general` ... `weakest_point`
Hard rules: say-not-know beats guess — "not-stated-in-paper", "cannot-reproduce-verbatim", `not_in_paper`: true, and `[]` are all HONEST outputs that score better than any confident fabrication. REVERSING a decision the paper actually made — claiming it chose X when it chose the opposite — is the single worst possible output.
digest/recall-scorecard.mdextractions/*.episodes.jsonVERIFIED-SOURCES.md| Slice | Paper | Entries | Confirmed | Partial | Not-in-paper | Contradicted | Inversions |
|---|---|---|---|---|---|---|---|
| L | Kessler, “The Alpha 21264 Microprocessor” · IEEE Micro 1999 | 57 | 72% | 16% | 7% | 5% | 0 |
| M | Hinton et al., “The Microarchitecture of the Pentium 4 Processor” · ITJ 2001 | 52 | 81% | 10% | 10% | 0 | |
| N | Warnock et al., “The circuit and physical design of the POWER4 microprocessor” | 48 | 75% | 8% | 17% | 0 | |
| O | Kongetira et al., “Niagara: A 32-Way Multithreaded SPARC Processor” | 59 | 80% | 3% | 17% | 0 |
| Model | Entries | Confirmed | Conf. rate | Part | NIP | Contra | Inv. | Slices |
|---|---|---|---|---|---|---|---|---|
| meta | 25 | 25 | 100% | 0 | 0 | 0 | 0 | all 5 |
| qwen | 14 | 14 | 100% | 0 | 0 | 0 | 0 | L M N O |
| sonnet | 12 | 12 | 100% | 0 | 0 | 0 | 0 | L M N O |
| grok | 25 | 23 | 92% | 1 | 1 | 0 | 0 | all 5 |
| chatgpt | 25 | 23 | 92% | 0 | 1 | 1 | 1 | all 5 |
| yuanbao | 24 | 22 | 92% | 1 | 0 | 1 | 1 | all 5 |
| gemini | 16 | 14 | 88% | 1 | 1 | 0 | 0 | L M N O |
| kimi | 19 | 15 | 79% | 2 | 1 | 1 | 1 | L M N O |
| deepseek | 25 | 20 | 80% | 2 | 2 | 1 | 1 | all 5 |
| mimo (prose) | 8 | 5 | 63% | 1 | 1 | 1 | 1 | D2 L only |
| minimax m3 | 16 | 6 | 38% | 6 | 1 | 3 | 3 | D2 L M O |
| minimax 2.7 | 24 | 9 | 38% | 8 | 5 | 2 | 2 | all 5 |
| glm 5.3 flash | 20 | 2 | 10% | 2 | 16 | 0 | 0 | L M N O |
metaglm 5.3 flashqwen confirmed 14 of 14 recall entries and then scored 2 of 3 SPFM vectors on a fabricated formula. meta is the only model strong at both: 25 of 25 on recall and 4 of 4 on the canonicalizer. The two competencies rank the field differently.grok scored 23 of 25 confirmed across all five slices, the best recall in the collection on the paper closest to the domain.paper_vs_general is informative but weak: entries the model marked from-general-knowledge are still 70% confirmed.