How a cell is falsified
The gates
A shadow-eval delta is never evidence by itself. Even with a proper chronological holdout and a frozen baseline, a headline number can be a trivial-rule match, leakage, or majority-class collapse. These are the checks a cell has to survive before the word promote is allowed.
The core set is { leakage, rule_parity, macro_f1_delta, bootstrap_ci_excludes_zero, mcnemar_p_lt_0.05 }. A verdict of promote requires every gate; anything else is a falsification, and a falsification updates the public record before any rerun.
Trivial-rule parity ceiling
The cell's macro-F1 must exceed a five-line task_class → lane lookup's macro-F1 by a margin that survives the bootstrap CI.
This is the trap V1 fell into. A lookup table fitted by majority vote per class can reproduce a +5 pp accuracy headline exactly. Run it before reporting any positive delta.
Evidence: V1: adapter 0.326 vs rule 0.326 — bit-identical.
Leakage detection
No prompt body in test may appear anywhere in train — byte compare plus hashed Jaccard. Report the raw duplicate count and the accuracy split between duplicate and clean rows.
Memorized rows print as skill. Dedup must be part of the split itself, not a post-hoc filter, or the split silently reintroduces it on the next rebuild.
Evidence: V1: 46/98 test bodies collided with train, 22/98 verbatim. The honest delta on the 76 clean rows was ≈ +2.6 pp, not +5.10.
Macro-F1 over accuracy
Report cell macro-F1 and trivial-rule macro-F1 side by side. If the difference is noise, the accuracy delta is noise too.
Accuracy is dominated by majority-class collapse; macro-F1 is class-balanced. A model that defaults to the most common training class prints positive accuracy deltas while never predicting some classes at all.
Evidence: V1: minimax predicted 72/98 times, claude recall ≈ 0.04, zai never predicted at all.
Bootstrap CI excludes zero
Resample the per-example correct/incorrect vector with replacement (N = 10,000) and require the CI95 on the accuracy delta to exclude zero.
A ±5 pp delta at n ≈ 100 has a confidence interval wide enough to span zero even when the point estimate looks clean.
Evidence: V1: CI95 was [−8.2, +18.4] pp.
McNemar exact, two-sided
p < 0.05 on the discordant pairs.
The paired test is what distinguishes a real per-example improvement from a reshuffle of the same error budget.
Evidence: V1: b=20, c=25 → p ≈ 0.55.
No hardcoded numbers
Grep the evaluator source for any literal float that could reach the report. Every baseline must be recomputed from real inference on the eval set.
A frozen baseline that is frozen because it is a constant is not a baseline.
Evidence: V1: the 47.96% baseline was a literal in the eval runner source.
No zero-recall class
Every provider class must have recall > 0.
Class collapse is the failure mode that a single aggregate metric is least able to see.
Evidence: V2 E4 collapsed to 0.00% accuracy after an 80× oversample of the rarest class.
Strict schema validity ≥ 95%
Output must validate against a schema whose enum values are derived from the corpus, not hardcoded — for the routing task, exactly four keys.
A permissive validity check that accepts any parseable dict measures nothing.
Evidence: V1 accepted any dict. V2 tightened it to exactly four enum-checked keys.
p95 latency ≤ 250 ms
Measured on the actual serving path, not on the training device.
A correct decision that cannot be made inside the request budget is not a routing policy. A latency violation is a structural signal to re-derive the architecture, not a prompt to shrink the model.
Evidence: V4 measured 1,635.9 ms at 3B and 954.08 ms at 1B. V5 measures 0.0773 ms.
Two independently frozen windows
A second evaluation window generated separately, with input values never seen in train, scored without retraining.
One frozen holdout proves the split was honest. Two prove the policy generalizes rather than interpolating the sampled grid.
Evidence: V5: train counts {0, 3, 6, 9}, OOD window counts {1, 2, 4, 5, 7, 8, 12} → 93.40% / 0.9260.
Beats the recovered teacher rule
Recover the deterministic policy the training labels came from, score it on the same test set, and require the learned model to beat it.
The audit-of-the-audit. A learned classifier that matches a deterministic teacher at 100% is not a shipping candidate — it is evidence the architectural ladder was the wrong choice and the rule is the actual artifact.
Evidence: V5: the recovered rule scores 100% against the model's 99.05%. The rule stays in production.
Three baselines, not one
V1's recipe called for a single frozen deterministic router. Every cell since requires three, because each answers a different falsification question and the audit checks all three at once.
| Baseline | What it answers | Why it is load-bearing |
|---|---|---|
majority_class | Is the cell learning anything at all? | The trap is class collapse. A model that always predicts the most common class prints a positive accuracy delta on a skewed corpus. This is the lower bound, and the cell must beat it on macro-F1, not accuracy. |
task_class_rule | Did the cell learn the trivial rule? | The structural trap V1 fell into. A task_class → mode lookup fits in five lines. This is the parity ceiling the cell must clear. |
tfidf_logreg | Is the learnable signal in the features? | A balanced sparse linear model answers whether there is learnable structure at all. If it beats the trained cell on macro-F1, the cell is failing to find structure a linear model already finds. Verified on V3: balanced TF-IDF reached macro-F1 0.7559 with 100% recall on the rarest class. |
Every matched-pairs evaluation emits all three under stable paths, with SHA-256 digests recorded in the manifest before the GPU job is submitted. The eval receipt's baseline block then carries three measurement anchors, and the audit has three arithmetic deltas to recompute against independent files.
Locking the window before training
The evaluation set is frozen before the training job is submitted, and the dataset manifest digest and base-model revision are captured in the receipt at that moment. Everything downstream compares against that pinned identity: if a later manifest carries a stale digest, the audit chain is broken and the receipt verifier says so rather than the result quietly meaning something else.
Splits are grouped by signature, never by row. A signature is built from the metadata — task class, delegation flag, lane shape, recent-count shape — and deliberately excludes the prompt body, so the same metadata cannot appear on both sides of the split. Deduplication happens inside the split, not as a filter applied afterwards, because a post-hoc filter silently reintroduces the leak on the next rebuild. When two rows share a signature but carry conflicting labels, the later timestamp wins and the older row is dropped: that preserves the question the router is actually being asked, which is what would I route now.
Determinism is asserted, not assumed. A rebuild has to produce byte-identical fixtures, checked with a digest manifest after every build, and the resulting label distribution has to match an independently computed expectation. When the two disagree, neither number is trusted until the disagreement is resolved — a split that matches by coincidence is a split nobody has checked.
Inference is greedy. Sampling makes a verdict non-reproducible across reruns, so decoding is deterministic wherever a decoder is involved at all.
Two independent auditors
- Dispatch at least two audit lanes from different model families, with read-only filesystem tools and a prompt that says recompute every claim from the raw artifacts, do not trust the parent's summary.
- A single auditor's 'looks plausible' is a single point of failure. Two auditors landing on the same wrong number means the verifier is broken, not the result.
- The convergence is the proof. In V1 a 337-line audit and a 257-line audit reached identical finding sets without seeing each other.
- When gates fail, correct the public record before doing anything else. Not 'rerun with tweaked hyperparameters', not 'ship anyway with caveats'.
Discipline
- Preregistration before training
- Dataset disclosure, metric set, baseline set, promotion gates and disallowed behaviors are written before a GPU job is submitted. When a planned baseline turns out to be unavailable, the substitution and its exact score go into the preregistration before the harness ships — a silent baseline swap is what an audit is for.
- Strict RED-GREEN-REFACTOR
- Every new behavior gets a failing test first, and the failing test has to fail for the right reason — feature missing, not a typo or a bad import path.
- Suspect the evaluator first
- When accuracy and schema validity both read zero and every baseline delta is negative with probability 1.0, the first hypothesis is a prompt-template or harness mismatch. The second is the adapter. The third is the dataset. Order matters: an evaluator misdiagnosis costs redundant GPU hours and contaminates the publication record.
- Falsification is a lesson, not a stop signal
- Correct the record, then iterate. A killed cell publishes as its own outcome and never overrides a prior audit's verdict.
- Public receipts carry no private text
- Raw generations live only in mode-0600 private artifacts. The public artifact strips prompts and completions and keeps the metrics, the provenance and the digests.