The first day with real ground truth. Every shown computation was re-executed independently in scratch Python, so a verdict here is a recomputation.
solar’s 17/19 is the largest set in the batch; mistral’s 3/11 is six substantive failures. * The free web chat build of GLM 5.3 Flash is the quantized one; the API-harness deployment of the same model name is a different build, and the two do not score alike.Each prompt pinned one regulatory instrument by its exact citation, supplied the decisive rule, and asked for worked cases with a weakest_point. Several prompts were built specifically to expose the place implementations err.
Show the TRID ten percent tolerance rule the Day 2 prompt pinned, and the wrong method it was built to catch.
A case that sums only the increases is the defect the prompt was built to expose. These numbers are illustrative; Day 2 recomputed about 720 real cases.
This task is about the TRID fee-tolerance rules — 12 CFR §1026.19(e)(3) — the zero-tolerance, 10%-aggregate-tolerance, and no-tolerance-limit buckets for comparing Loan Estimate to Closing Disclosure charges.
Produce up to 5 worked tolerance-test cases, each with a fee table (LE amount vs CD amount per fee, bucket assignment per the rule) — and at least 2 cases MUST have mixed directions inside the 10% bucket (some fees increased, some decreased), because the 10% test compares AGGREGATE to AGGREGATE and mixed-direction cases are where implementations get it wrong.
...`computation`: step by step — per-bucket sums, the 10% threshold (Σle × 0.10), the aggregate comparison (Σcd vs Σle, NOT the sum of increases) ... `weakest_point`
Hard rules: the load-bearing rule is aggregate-vs-aggregate for the 10% bucket — a case that computes the sum-of-increases instead is itself the defect this suite exists to catch.
Families and the rule each one turns on:
digest/day2-c1-adjudication.mdstatistics.NormalDist; a bisection APRC solver; a scratch RFC 8785 canonicalizer self-tested against the RFC's own number vectors.digest/day2-c1b-c2-adjudication.md§2 of both adjudications| Model | Verified | Self-inconsistent | Wrong-input | Wrong-rule | Anti-reversal | Class-1 verdict | |
|---|---|---|---|---|---|---|---|
| chatgpt | 15/15 | 0 | 0 | 0 | 0 | EXEMPLARY | APPJ rates match independent bisection to 1e-10 across four code paths; 5/5 GLOBE examples close. One wrong-content file (OFAC = its TRID) was not assessable |
| kimi | 20/20 | 0 | 0 | 0 | 0 | EXEMPLARY | Clean on all four prompts; reproduces both published APPJ anchors |
| grok | 20/20 | 0 | 0 | 0 | 0 | EXEMPLARY | Both APPJ anchors, full-attribution OFAC |
| meta | 18/18 | 0 | 0 | 0 | 0 | EXCELLENT | Labels all APPJ cases truthfully as constructed |
| qwen | 19/19 | 0 | 0 | 0 | 0 | EXCELLENT | Contributes the sharpest TRID discriminator (+400/−400, aggregate zero) |
| sonnet | 18/18 | 0 | 0 | 0 | 0 | EXCELLENT | The only 47% near-miss OFAC vector |
| minimax 2.7 | 19/19 | 0 | 0 | 0 | 0 | EXCELLENT | See the reinstatement below |
| yuanbao | 19/20 | 0 | 0 | 0 | 1 | STRONG | Form 1 and Form 3 closed forms verify exactly; one TRID case titled “aggregate passes” while its own verdict is a violation |
| glm 5.3 flash | 14/14 | 0 | 1 (GLOBE rate year) | 0 | 0 | STRONG | Best OFAC gate case in the batch; its trid file is absent |
| mimo | 19/20 | 1 (OFAC) | 0 | 0 | 0 | GOOD | Produced parseable JSON for the first time in the collection |
| gemini | 15/16 | 0 | 0 | 1 (OFAC multiplicative) | 0 | GOOD (with a rule defect) | Arithmetic exact, legal treatment wrong on the one case the prompt was built to expose |
| deepseek | 18/20 | 1 (TRID) | 1 (GLOBE rate pair) | 0 | 0 | MIXED | Substituted its own inputs mid-computation to manufacture a violation |
| mistral | 17/20 | 3 (OFAC) | 0 | 0 | 0 | MIXED | Correct attribution arithmetic; blocked-entity verdicts omit the intermediaries |
| minimax m3 | 3/3 recovered | 0 | 0 | 0 | 0 | NOT ASSESSABLE | Token dropout in 3 of 4 files |
| Model | IRB | FRTB | MICA | CCD2 | JCS | Batch-2 verdict | |
|---|---|---|---|---|---|---|---|
| chatgpt | 4/4 | ✓ | ✓ | 4/4 | 10/11 | EXEMPLARY | IRB to 15 s.f.; JCS recovered only by salvage (its inner quotes were unescaped) |
| kimi | 4/4 | ✓ | ✓ | absent | 12/12 | EXEMPLARY | The only model that derives its Basel benchmark (CRE99.2 at LGD 40%, scaled) rather than asserting one; canonicalizer exact on all 12 |
| sonnet | 4/4 | ✓ | ✓ | 4/4 + 1 truthful UNKNOWN | 12/12 | EXEMPLARY | Declines to assert an APRC it cannot close; joint-best JCS |
| meta | 4/4 | ✓ | ✓ | ✓ | 10/10 | EXEMPLARY | The only receipt file with source-level detail: domain-separator tags, 4-byte selector, 128-bit claim-digest split |
| grok | 4/4 | ✓ | ✓ | 5/5 | 10/11 | EXCELLENT | One escaping-form miss only |
| qwen | 4/4 | ✓ | ✓ (4 UNKNOWN flags) | 5/5 | 9/10 | EXCELLENT | Its MiCA flags sit on the one figure now marked NEEDS-RETRIEVAL |
| mimo | absent | ✓ | ✓ | 5/5 | 9/11 | STRONG | Second consecutive batch of parseable, correct output |
| glm 5.3 flash | – | – | – | – | 9/11 | INSUFFICIENT DATA | 2 of 6 files. One anti-reversal instance |
| yuanbao | WRONG-RULE 4/4 | ✓ | ✓ | 5/5 | 9/11 | SPLIT | Exact on APRC, wrong grouping on CRE31 |
| deepseek | 4/4 | ✓ | ✓ | SELF-INCONSISTENT 3 | 13/15, 1 WRONG-RULE | MIXED | The batch’s only UTF-16 key-order failure; APRCs disconnected from its own schedules |
| gemini | 3/4, 1 SELF-INCONSISTENT | ✓ | ✓ | SELF-INCONSISTENT 3 | 6/8 | MIXED | Nominal rate reported as APRC; fabricated a benchmark that confirms its own error |
| solar | WRONG-RULE 4/4 | ✓ | ✓ | 1/1 | 17/19 | SPLIT | Best JCS volume in the batch, worst IRB formula |
| minimax 2.7 | WRONG-RULE 4/4 + WRONG-INPUT | ✓ | ✓ | WRONG-RULE 2 | 5/10 | WEAK but self-flagged | Four independent IRB defects, each correctly reported as not matching benchmark |
| minimax m3 | SELF-INCONSISTENT | ✓ | ✓ | SELF-INCONSISTENT 3 | 6/9 | WEAK | Fabricated a BCBS table citation to explain away its own arithmetic |
| mistral | WRONG-RULE 4/4 | ✓ | ✓ | WRONG-RULE + SELF-INCONSISTENT 3 | 3/11 | BOTTOM | Weakest on arithmetic, rules and output integrity simultaneously |
| Model | OK | Form | Wrong | Score | Note |
|---|---|---|---|---|---|
| solar | 17 | 2 | 0 | 17/19 | Largest set (21 vectors); both misses are escaping form |
| deepseek | 13 | 0 | 2 | 13/15 | The only key-order failure in the batch |
| kimi | 12 | 0 | 0 | 12/12 | Correct on all 12 |
| sonnet | 12 | 0 | 0 | 12/12 | Correct on all 12 |
| chatgpt | 10 | 1 | 0 | 10/11 | Recovered only by salvage |
| grok | 10 | 1 | 0 | 10/11 | |
| meta | 10 | 0 | 0 | 10/10 | One vector has an empty input_json; its stated output for it is correct |
| mimo | 9 | 2 | 0 | 9/11 | |
| yuanbao | 9 | 2 | 0 | 9/11 | |
| qwen | 9 | 0 | 1 | 9/10 | -0 preserved |
| glm 5.3 flash | 9 | 0 | 2 | 9/11 | |
| gemini | 6 | 2 | 0 | 6/8 | |
| minimax m3 | 6 | 0 | 3 | 6/9 | 2 vectors have malformed input_json |
| minimax 2.7 | 5 | 2 | 3 | 5/10 | |
| mistral | 3 | 2 | 6 | 3/11 | Six substantive failures spanning every rule family |
kimimistralgemini gets the OFAC arithmetic exactly right and applies the US multiplicative treatment where the EU attribution test applies, on the one case the prompt was built to expose. grok is 20/20 on Class-1 and invents an EAR band elsewhere. Both close arithmetically, so a recomputation gate passes both.yuanbao multiplies where the grouping requires division. solar omits the /√(1−R) term entirely. minimax 2.7 multiplies and carries three further defects. mistral drops the /(1−1.5b) divisor from the formula itself. Four spellings of one structural error, and all four close on their own inputs.weakest_point was populated on 55 of 55 files and on every item. About 40% name the true fault line: gemini’s OFAC entry names the exact axis on which it is wrong, and deepseek flags the rate year that is the wrong constant. About 45% are real but secondary and 15% are boilerplate. It still missed the one defect that mattered, deepseek’s TRID case 2, which does not mention its own input substitution.minimax m3 loses a key mid-object: trid: {"name": …, "bucket": "ten_pct_aggregate", 500.00, "cd_amount": 350.00}, where the "le_amount": key has vanished. Recorded as damaged delivery rather than as a wrong answer, and the three recoverable entries were graded.