The experiment ledger

Why this stage ran

The headline delta looked promotion-grade: the eval runner exited 0, every receipt field was populated, the pipeline wrote the public site, and the Pages deploy succeeded.

Why it did not advance

Two independent audit lanes (Grok 4.5, 337 lines; Kimi K3, 257 lines) recomputed every claim from the raw artifacts with read-only tools and converged on identical findings without seeing each other. The delta was a trivial-rule match sitting on top of leakage and majority-class collapse.

Measurements

Metric	Value	Note
Adapter accuracy	53.06%	n=98
Baseline accuracy	47.96%	hardcoded literal, not recomputed
Adapter macro-F1	0.326	bit-identical to the 5-line rule
Rule macro-F1	0.326	task_class → lane lookup
Macro-F1 delta	0.0007	rule parity
Bootstrap CI95 (accuracy delta)	[−8.2, +18.4] pp	spans zero
McNemar exact two-sided	p ≈ 0.55	b=20, c=25 (p=0.5515)
Honest delta on deduped rows	≈ +2.6 pp	76 clean rows after removing collisions
Train/test byte collisions	46 / 98	22 of them verbatim training strings

Gate results

Gate	Bar	Measured	Result
Trivial-rule parity	adapter macro-F1 > rule macro-F1 by a margin surviving the bootstrap CI	adapter 0.326 / rule 0.326 — bit-identical	FAIL
Leakage-free holdout	zero prompt bodies collide with train	46/98 byte-collide; 22/98 verbatim train strings	FAIL
Macro-F1 over accuracy	macro-F1 delta with CI excluding 0	delta 0.0007; accuracy CI spans zero	FAIL
Bootstrap CI significance	CI95 on per-example accuracy delta excludes 0	[−8.2, +18.4] pp	FAIL
McNemar exact two-sided	p < 0.05 on discordant pairs	b=20, c=25 → p ≈ 0.55	FAIL
No majority-class collapse	per-class recall ≥ ε for every class	minimax predicted 72/98; claude recall 0.04; zai never predicted	FAIL
No hardcoded numbers	baseline recomputed from real inference	47.96% literal in the eval runner source	FAIL

The source reference records the verdict as 5/7 gates failed → falsified, while its per-check table marks all seven checks failing. Both statements are reproduced here rather than reconciled, because reconciling them would mean inventing a number.

Receipt-level defects the audit surfaced

• valid_json accepted any parseable dict — no schema check on the required keys.
• Raw generations were never stored; only the parsed object was persisted.
• The keyword-baseline fit() was a no-op, so the embedding baseline never ran.
• The 972-row experiment was not covered by the preregistration.
• The chronological split was mechanically correct but semantically broken by drift: after dedup and latest-wins, test flipped to 7% minimax from 72% in train.

Public-record correction

When promotion gates fail, the response is not a rerun with tweaked hyperparameters and not shipping with caveats — it is correcting the public record first.

Field	Value
branch	fix/router-school-audit-verdict
head	1083652a6e38e1f06c9515f81f8389abd161528b
merge commit	e5b2dc4144aa49ebf5adc976f9b06dd6450aca08
pipelines	15881 (MR) and 15882 (main post-merge) — all jobs success with allow_failure=false
live read-back	version 0.2.1 · station status failed · campaign 3 completed / 1 failed

Artifacts

Artifact	Path	Visibility	SHA-256	Note
Public site record	data/projects/router-school.json	public	not published in this record	no private prompt text
Row-level eval output	tooling/artifacts/router-school-eval.json	private	withheld — private artifact	mode 0600 — body withheld
Independent audit transcripts	Grok 4.5 (337 lines) · Kimi K3 (257 lines)	private	withheld — private artifact	dispatch-local, bodies withheld

Why this stage ran

V1's lesson was not 'train one more epoch'. Every V1 defect was closed structurally before a single GPU cell was scheduled: dedup moved into the split itself, baselines recomputed from live inference, raw generations persisted, strict schema validation, and a preregistration written before training.

Why it did not advance

All three GPU cells failed the promotion gate. None beat the modal task_class → lane rule (macro-F1 0.2351) by a margin whose CI excluded zero. The matrix was the right experiment and it correctly killed V2.

Foundation rebuild

Test suite. 65 → 87 green (+22 across seven RED-GREEN slices; no parent test weakened or skipped)

V1 defect	V2 fix
972 raw rows with no dedup	972 → 601 unique signatures
Conflicting duplicate rows kept verbatim	26 conflicting signatures / 167 rows resolved latest-wins
Cross-split leakage possible	0 cross-split signature collisions, asserted by test
Random split with no signature grouping	signature-grouped split 480 / 60 / 61
Test label distribution not asserted	independently matches the audit: claude 21 / grok 21 / zai 9 / kimi 6 / minimax 4
No determinism guarantee	deterministic rebuild test plus sha256sum -c after every build

strict valid_json requires exactly four keys — task_class, recommended_lane, model, confidence — with enum values derived from the corpus, not hardcoded

Measurements

Metric	Value	Note
Majority baseline	acc 0.0656 / macro-F1 0.0246	61 test rows
Task-class rule baseline	acc 0.3115 / macro-F1 0.2351	the parity ceiling
TF-IDF + logreg baseline	acc 0.3115 / macro-F1 0.2351	runtime 0.025 s; substituted for the unavailable embedding baseline and recorded in the preregistration before the harness shipped
E2 — V1 adapter on V2 holdout	acc 18.03% / macro-F1 0.1297 / JSON 52.46%	worse than the rule on every class
E3 — fresh rank-16 QLoRA (r=16, α=32)	acc 29.51% / macro-F1 0.1905 / JSON 96.72%	within noise of the rule
E4 — class-balanced rank-32 (80× zai oversample)	acc 0.00% / macro-F1 0.0000 / JSON 18.03%	generation collapse

Gate results

Gate	Bar	Measured	Result
Adapter macro-F1 > task-class rule	by ≥ CI half-width	best cell 0.1905 vs rule 0.2351	FAIL
Bootstrap CI95 on accuracy delta	excludes 0	no cell cleared it	FAIL
McNemar exact two-sided	p < 0.05	no cell cleared it	FAIL
All five providers have recall > 0	yes	E4 collapsed entirely	FAIL
Strict JSON validity rate	≥ 95%	E2 52.46% · E3 96.72% · E4 18.03%	FAIL

Diagnostic

Artifacts

Artifact	Path	Visibility	SHA-256	Note
V2 baselines	tooling/artifacts/router-school-v2-baselines.json	public	not published in this record	majority + task-class rule
V2 TF-IDF logreg baseline	tooling/artifacts/router-school-v2-tfidf-logreg.json	public	not published in this record	balanced word 1–2gram
V2 fixtures + manifest	tooling/fixtures/v2/{train,val,test,manifest}.json	public	not published in this record	deterministic rebuild verified by sha256sum -c
Preregistration	tooling/ROUTER_SCHOOL_V2_PREREGISTRATION.md	public	not published in this record	P1–P9 gates written before training

Why this stage ran

V2 proved the dataset itself was the wrong signal. The production label is a function of (prompt, recent 24h provider counts) — the dispatch guard's portfolio-balancing sort. V2 dropped that history field, so identical prompts carried conflicting labels resolved by latest-wins, and the adapter learned 'the most recent conflicting label'. V3 makes the hidden variable explicit and trains against the deterministic teacher the live router actually runs, so there is no drift between training and serving.

Why it did not advance

The evaluator built the generation prefix twice, appending a second response marker after the instruction template already ended with one. Every adapter scored through that harness emitted a scalar cascade instead of JSON. The fingerprint — accuracy 0 and json_validity 0 and every baseline delta ≤ 0 with probability 1.0 — is the signature of a prompt-template mismatch, not a bad adapter.

Dataset shape

Field	Value
static cells	810 — 10 task classes × 3 steps × 3 lanes × 3 files × 3 minutes
fairness cells	500 background cells sampled from a 1,024-state recent-count grid over {0, 3, 6, 9}, shuffled with a fixed seed
total	1,310 unique examples after dedup by text hash
inputs	task_class (10 enum values), must_delegate, lanes, estimated steps / files / minutes, and recent_<provider> counts for claude, grok, kimi, minimax, zai
schema	owlforge.router-school.policy-distill.v3

Measurements

Metric	Value	Note
Balanced TF-IDF on the V3 test split	macro-F1 0.7559	131 examples, 100% zai recall — the fairness-state features do carry learnable signal
First adapter eval (invalidated)	accuracy 0.0 / macro-F1 0.0 / JSON validity 0.0	harness defect, not model quality
Harness test suite after the fix	104 → 105 green	new test asserts exactly one response marker

Gate results

Gate	Bar	Measured	Result
Evaluator measures the model, not itself	exactly one response marker in the inference prompt	two markers — every score through this harness was uninformative	FAIL

The fix

A single canonical inference_prompt() helper returning the exact prefix with one marker and a trailing space, landed RED-GREEN with a test that asserts the marker count. Three already-trained adapters were re-queued for re-evaluation without retraining — the adapters were never at fault, so no additional GPU hours were spent on identical weights.

Diagnostic

Artifacts

Artifact	Path	Visibility	SHA-256	Note
V3 train SFT manifest	tooling/fixtures/v3-policy-distill/	public	b6de0235529d23f47a42d0dcbc6bc368b373b68159e95a0c314738121086c864	manifest SHA-256 of the train SFT split
V3 policy-distill eval (invalidated)	tooling/artifacts/router-school-v3-policy-r16-eval.json	public	not published in this record	retained as the record of the harness defect

Why this stage ran

V3 showed the fairness surface was the load-bearing signal and that it was too thinly sampled to learn. V4 isolates it: a stratified grid over the five-provider recent-count state space with a fixed task_class and a fixed must_delegate flag, so the student learns the portfolio-balancing decision without the noise of fit-score dispatch on unrelated task classes. Stride sampling rather than random shuffle keeps edge states — one provider at 12, the rest at 0 — represented in the test split instead of clustering it around the modal tier.

Why it did not advance

The registered p95 latency target is 250 ms. V4 measured 1,635.9 ms — 6.5× over budget. The fairness decision was correct and the adapter was too slow to run in the live router.

Measurements

Metric	Value	Note
Accuracy	68.57%	105-example test set
Macro-F1	0.6912	CI95 0.5886–0.7734
Weighted-F1	0.6867
Strict JSON validity	100%
p95 latency	1,635.9 ms	budget 250 ms — 6.5× over
Runtime	305.554 s	peak util 92% · peak VRAM 3,180 MiB · peak power 332.8 W · peak temp 68 °C
Majority baseline	acc 0.3429 / macro-F1 0.1021	collapses to MiniMax; the task-class rule is identical because V4 fixes task_class
TF-IDF + balanced logreg baseline	acc 0.1333 / macro-F1 0.0471	collapsed entirely to Claude — sparse word features cannot generalize a numeric ordering

Per-class results

Provider	Recall	F1
Claude	92.86%	0.8387
Grok	47.37%	0.6429
Kimi	70.00%	0.5833
MiniMax	66.67%	0.6154
Z.ai	73.08%	0.7755

Gate results

Gate	Bar	Measured	Result
Adapter macro-F1 > task-class rule	by ≥ CI half-width	0.6912 vs 0.1021 (CI half-width 0.0924)	PASS
Bootstrap CI95 on accuracy delta	excludes 0	[0.6, 0.7714]	PASS
McNemar exact two-sided	p < 0.05	1.06e-16 vs TF-IDF · 3.18e-6 vs majority	PASS
All five providers have recall > 0	yes	minimum 47.37% (Grok)	PASS
Strict JSON validity rate	≥ 95%	100%	PASS
p95 latency budget	≤ 250 ms	1,635.9 ms	FAIL

Artifacts

Artifact	Path	Visibility	SHA-256	Note
V4 dataset manifest	tooling/fixtures/v4-fairness-specialist/manifest.json	public	e082229bab312618f4d4e4c1590dcee3951b03d7734bc45d1c2af8d0e9dab8d7	schema owlforge.router-school.fairness-specialist.v4
V4 train SFT split	tooling/fixtures/v4-fairness-specialist/train-sft.json	public	5abcd931c25d8a56ca73cc6cfba4f8f167f8557f8e0fe5436b2fb0e59d86201a
V4 train split	tooling/fixtures/v4-fairness-specialist/train.json	public	d10b425b4bbd706ecf3d8f722ebe4c60c82744987424de42342bb03587107cac
V4 test split	tooling/fixtures/v4-fairness-specialist/test.json	public	1e9b03dca19423b08ed1396b3b3ae3999813a2b3869edcc48946a1fb9534f037	105 rows — the frozen window every later cell is scored on
V4 adapter	artifacts/checkpoints/router-v4-fairness-r16-lora	public	b44784f8f0283fcb2169b418de7d10dc186dc80277f663e6bd81c9d466e89837	adapter SHA pinned in both the train and eval wrappers
V4 baselines	tooling/artifacts/router-school-v4-baselines.json	public	not published in this record	written before the GPU cell shipped

Why this stage ran

V4's quality was promotion-grade and only latency blocked it, so the registered next arm was the obvious one: hold the curriculum, shrink the base model, and check whether the fairness signal survives a 3× parameter reduction.

Why it did not advance

It did not survive. The 1B adapter defaulted to emitting raw scalar tokens for almost every prompt; only 14 of 105 test rows produced parseable JSON. It never learned the fairness policy — it learned that emitting a scalar is the lowest-loss response.

Measurements

Metric	Value	Note
Accuracy	13.33%	vs 68.57% at 3B — −55.24 pp
Macro-F1	0.1292	vs 0.6912 at 3B — −0.5620
Strict JSON validity	13.33%	vs 100% at 3B — −86.67 pp · 14 of 105 rows parseable
p95 latency	954.08 ms	1.6× faster than 3B, still 3.8× over the 250 ms budget
Runtime	110.253 s	2.77× faster · peak VRAM 1,790 MiB (43.7% lower) · peak power 315.81 W · peak temp 65 °C
Adapter size	45,118,424 bytes
Cost per cell	1.8 min	vs 5.1 min at 3B — 2.8× cheaper and worthless

Gate results

Gate	Bar	Measured	Result
Adapter macro-F1 > task-class rule	by ≥ CI half-width	0.1292	FAIL
Bootstrap CI95 on accuracy delta	excludes 0	not cleared	FAIL
McNemar exact two-sided	p < 0.05	not cleared	FAIL
All five providers have recall > 0	yes	kimi never predicted; 91 of 105 rows unparseable	FAIL
Strict JSON validity rate	≥ 95%	13.33%	FAIL
p95 latency budget	≤ 250 ms	954.08 ms	FAIL

Diagnostic

Artifacts

Artifact	Path	Visibility	SHA-256	Note
V4-1B adapter	artifacts/checkpoints/router-v4-fairness-1b-r16-lora	public	73fb4805a9ea32aa210bf60074f719317d8f8bfed13517d985ca7cb34e3887f3
V4-1B public eval	tooling/artifacts/router-school-v4-fairness-1b-r16-eval.json	public	not published in this record
V4-1B private eval	tooling/artifacts/router-school-v4-fairness-1b-r16-eval-private.json	private	withheld — private artifact	raw generations — body withheld

Why this stage ran

V4-1B closed the generative ladder. For a five-class numeric classification over five counters the correct curriculum is rule → sparse linear → tiny MLP → boosted trees → generative decoder, and classification starts at the third rung. V5 picks that rung on the identical dataset, and the output is a class label rather than a generated document.

Measurements

Metric	Value	Note
Accuracy	99.05%	104 / 105
Macro-F1	0.9831
Weighted-F1	0.9905
Strict JSON validity	100%	no JSON is generated — the class is the output
Bootstrap accuracy CI95	97.14–100%
Bootstrap macro-F1 CI95	0.9398–1.0
Stdlib CPU p95	0.0773 ms	the deployment target — ~12,000× faster than the failed 1B decoder and ~21,000× faster than the 3B QLoRA adapter
Torch CPU p95	0.0216 ms
CUDA p95	0.1841 ms
CUDA training time	12.157 s	selected epoch 3
Parameters	1,573	model size 50,765 bytes

Two independently frozen windows

Train counts {0, 3, 6, 9}; held-out window counts {1, 2, 4, 5, 7, 8, 12}. 500 cells, generated separately, no retraining.

Metric	Value	Note
OOD accuracy	93.40%
OOD macro-F1	0.9260
Improvement over majority-class baseline	CI95 +0.514 to +0.612

The learned numeric policy survives a genuinely different count distribution and clears the two-independently-frozen-windows requirement.

The audit-of-the-audit

A 99.05% headline is meaningless until the student is shown to match a known teacher. The honest audit recovered the deterministic teacher policy from the dispatch guard.

1. Select the provider with the lowest recent dispatch count.
2. Break ties in priority order: MiniMax → Z.ai → Grok → Claude → Kimi.

That rule scores 100% on the same 105-example test set.

Gate results

Gate	Bar	Measured	Result
Accuracy / macro-F1 over every baseline	CI excludes 0	99.05% / 0.9831 with accuracy CI95 97.14–100%	PASS
All five providers have recall > 0	yes	104 of 105 correct across the five-class surface	PASS
Strict validity rate	≥ 95%	100% — no generation step to be invalid	PASS
p95 latency budget	≤ 250 ms	0.0773 ms on the stdlib CPU serving path	PASS
Two independently frozen windows	generalizes to unseen counter values	93.40% / 0.9260 on a 500-cell OOD window	PASS
Beats the recovered deterministic teacher	strictly above the rule	rule 100% vs model 99.05% — the rule wins	FAIL

Independent verification

• 64 / 64 targeted tests and 169 / 169 full-suite tests green.
• The job manifest was accepted on the first submission with no missing-key errors.
• Torch and stdlib forward passes produced identical predictions on all 105 test rows; maximum logit difference 3.09e-7.
• The GPU job completed in 33.122 s with no timeout, tracker error, or capacity issue.
• Artifact modes enforced: 0600 on the model JSON and the private evaluation, 0664 on the public evaluation.
• Public eval, private eval and model JSON each carry a matching SHA-256 in the receipt's provenance list.
• The experiment tracker records the run with status success.

What this cell does not close

• It does not produce a learned policy that beats the deterministic rule.
• It does not generalize to task classes other than background.
• It does not prove the rule is the best policy for any future routing change — only for the current production policy.

Artifacts

Artifact	Path	Visibility	SHA-256	Note
V5 public evaluation	tooling/artifacts/router-school-v5-fairness-mlp-eval.json	public	d4f0b8f52d000f8bbb8432a962db3e62cf238cc94e50dc1de94cad6e0f74590d	mode 0664 — public by design
V5 private evaluation	tooling/artifacts/router-school-v5-fairness-mlp-eval-private.json	private	withheld — private artifact	mode 0600 — row-level bodies withheld; digest held in the private receipt
V5 model	tooling/artifacts/models/router-v5-fairness-mlp.json	private	withheld — private artifact	mode 0600 — digest held in the private receipt
V5 OOD window	tooling/artifacts/router-school-v5-ood-window.json	public	not published in this record	a private companion window is held at mode 0600
V5 compute receipt	gpu-lab/receipts/20260804T144750Z-router-school-v5-fairness-mlp.json	public	not published in this record	provenance.artifacts carries the matching digest for every artifact above
Experiment-tracker run	mlflow run 7c1d3fdd4470470eba842f7bab883ec2	public	not published in this record	status success