What the failures established

The opening result of this programme was a clean-looking +5.10 pp accuracy delta against a frozen deterministic baseline, on a chronological holdout, with a fully populated receipt and a green pipeline. It was wrong in four independent ways at once. The adapter's macro-F1 was bit-identical to a five-line lookup table. Forty-six of ninety-eight test prompts collided byte-for-byte with training rows and twenty-two were verbatim training strings. One class was predicted for three quarters of the test set while another was never predicted at all. And the baseline it was being compared against was a literal constant in the evaluator source, not a recomputation.

None of those defects is visible in the headline number. All four are visible the moment somebody recomputes the claim from the raw artifacts. The generalizable form: a metric is a claim about a computation, and until the computation is reproduced by someone who did not perform it, the metric is a hypothesis.

Two audit lanes from different model families were given read-only filesystem access and one instruction: recompute every claim from the raw artifacts, do not trust the parent's summary. They produced a 337-line and a 257-line audit respectively, without visibility into each other, and landed on identical finding sets.

That convergence is what made the falsification trustworthy — not either audit on its own. A single reviewer saying looks plausible is a single point of failure with an authority signal attached. The inverse case is equally informative: when two independent auditors converge on the same wrong number, the verifier they share is broken, not the result.

When gates fail, there are three tempting responses and all of them are wrong: rerun with tweaked hyperparameters until a split cooperates, ship the result with caveats attached, or quietly stop publishing. The response that was actually taken was to correct the public record first — a merge request that flipped the published status to failed, carried the full falsification note, and was verified by reading the live site back after deploy.

The ordering is the point. Once a corrected record exists, the next experiment is an experiment. Before it exists, the next experiment is a search for a number that makes the old claim retroactively true, which is a different activity wearing the same clothes.

The V2 comparison between the old adapter re-scored on the new holdout and a freshly trained adapter on the cleaned dataset is the most useful single diagnostic in this programme. Accuracy rose 11.5 points. Schema validity rose 44 points. Macro-F1 did not move outside noise of the trivial rule.

Read carefully, that says the dataset cleanup repaired the model's ability to emit a well-formed answer and changed nothing about which answer it chose. A rise in aggregate accuracy that is not accompanied by a rise in class-balanced score is very often a formatting repair being mistaken for a capability gain.

The production label was never a function of the prompt alone. It was a function of the prompt and the recent per-provider dispatch counts, because the dispatcher balances a portfolio. The V2 dataset dropped that state, so two identical prompts could carry different labels, and the conflict was resolved by keeping the most recent one. The model was therefore trained to predict "the most recent conflicting label" — a noisy proxy for a decision made at a frozen moment.

No amount of rank, epochs or class balancing recovers a variable that is absent from the input. The check that catches this is cheap and worth running before any training matrix: take the teacher function, list every argument it actually reads, and confirm each one is present in a row. When a training corpus contains contradictory labels for identical inputs, that contradiction is the signature of a missing feature, not of noisy annotation.

A V3 evaluation returned zero accuracy, zero schema validity, and a negative delta against every baseline with probability 1.0. The obvious reading is a catastrophically bad adapter. The actual cause was that the harness appended a response marker to a prompt template that already ended with one, so every model scored through it emitted a degenerate cascade instead of an answer.

The three conditions firing together are a fingerprint, and the correct hypothesis order is: the evaluator, then the adapter, then the dataset. A harness misdiagnosis costs redundant GPU hours and contaminates the publication record; a five-minute prompt diff costs nothing. The durable fix is not the patch — it is the test that asserts the inference prefix contains exactly one response marker, so the next refactor cannot silently reintroduce it.

The corollary that saved real compute: when the harness was at fault, the already-trained adapters were re-scored rather than retrained. The weights were never in question.

V4 passed every quality gate and missed the latency budget by 6.5×. The registered next move was the intuitive one: hold the curriculum, shrink the base model, recover the budget. It cost 55 points of accuracy and 87 points of schema validity to buy a 1.6× latency improvement that was still nearly four times over budget. The smaller decoder did not learn a smaller version of the policy; it learned that emitting a bare scalar was the lowest-loss response.

The reason is structural rather than incidental. The task's output is one class label out of five. An autoregressive decoder cannot produce that label without generating an entire document around it, so the floor on its latency is set by the document, not by the decision. Shrinking the model moves the floor a little and destroys the decision.

Stated generally: when a learned component is too slow for its budget, the next arm is not a smaller version of the same architecture — it is re-deriving the architecture from the shape of the task. The distillation ladder of lower rank, smaller base, soft labels is valid only when the task is genuinely generative.

Once the task was correctly identified as five-class numeric classification over five counters, the architecture followed mechanically. The classification ladder runs rule, sparse linear, tiny MLP, boosted trees, generative decoder — and classification tasks start at the third rung, climbing left only if the small model fails. Generative tasks start at the far end and climb down.

Applying that rule at the start of the programme would have skipped four stages of decoder work. The 1,573-parameter network that resulted trains in about twelve seconds, serves in well under a tenth of a millisecond on a plain CPU path, and is roughly four orders of magnitude faster than either decoder attempt. The classifier was never the hard part; choosing to build a classifier was.

The reflex this programme runs on is correct the record, then iterate. A killed cell is published with its own outcome and never overwrites an earlier verdict; V1's falsification stands beside V5's result rather than being replaced by it. That is what makes the ledger append-only in a meaningful sense rather than a structural one.

It also changes what a stage is allowed to cost. Because a failure is publishable, a stage can be bounded aggressively — a small matrix, a frozen window, a preregistered gate set — instead of being run until it produces something worth announcing. Four falsifications in a single programme is not a poor record. It is what a working grading function looks like from the inside.