The Experiment We Refused to Run

How a pre-label audit stopped 2.5 billion engine nodes from answering the wrong question

The dangerous moment in an experiment is not always when the result appears. Sometimes it is the moment just before an expensive job starts, when the code looks complete, the sample is frozen, and everyone is tired of designing.

We reached that moment with A110B, our retrospective attempt to separate two sources of chess error:

• proposal regret: how much value was lost because the displayed candidate set omitted the best legal move; and
• selector regret: how much value was lost because the chosen move was worse than the best displayed candidate.

The first design was coherent, content-addressed, and tested. It froze 1,000 Luna response attempts covering 947 unique positions and 24,915 legal root moves. At 100,000 Stockfish nodes per move, it requested at most 2.4915 billion nodes. It had a live-process guard, atomic cache writes, source-game bootstrap intervals, and a release manifest.

We did not run it.

An adversarial audit found that several details could make the output precise, reproducible, and wrong.

The question looked simple

For each historical decision, let:

• V* be the evaluator value of the best legal move;
• VP be the value of the best displayed proposal; and
• VS be the value of the proposal-arm action.

Then:

proposal regret = V* - VP
selector regret = VP - VS
total regret    = V* - VS

The decomposition is additive by construction. If proposal regret dominates, we should study policy recall and nested candidate generation. If selector regret dominates, we should study comparison, order, and search. If both are material, we should cross proposer and selector interventions.

That logic remains useful. The problem was not the algebra. It was what the symbols actually measured.

Bug one: the WDL was not the WDL we named

The runner enabled Stockfish's UCI_ShowWDL option, but then ignored the engine-reported info wdl field. Instead it converted the centipawn score to WDL with python-chess's Score.wdl(model="sf") helper.

That sounds equivalent. It is not.

The helper applies a library calibration model. In the installed python-chess version, the generic sf route did not mean “use the WDL emitted by this exact Stockfish 18 binary.” A result labeled Stockfish 18 WDL could therefore have been a library-derived approximation calibrated for a different Stockfish generation.

The fix was explicit:

root_color = root_board.turn
info = engine.analyse(root_board, root_moves=[move], ...)
wdl = info["wdl"].pov(root_color)

The schema now binds wdl_source = engine_reported. Missing engine WDL is a hard failure. Tests use asymmetric WDL values for White and Black, plus mate, stalemate, and nonterminal positions, so a perspective inversion cannot hide behind a symmetric fixture.

This was not a cosmetic implementation correction. A sign error or mislabeled calibration could reverse the regret decomposition while every hash and cache check remained green.

Bug two: “direct” did not mean one thing

The frozen sample originally described 800 rows as model-direct selections and 200 as system fallbacks. The audit split those 800 rows:

Parser source	Rows
Accepted make_move command	433
Last displayed legal UCI token	367
Lexicographic system fallback	200

A last-token recovery is useful operational behavior, but it is not equivalent to an explicit command. Pooling them under “the model selected” would turn a parser policy into a latent-intent claim.

Even accepted commands are only an operational proxy for intent. A response can contain more than one command or mention a different legal action later. The amended frozen plan therefore audits every accepted response. Of 433 accepted-command rows, 432 contain exactly one command and no later distinct legal action. One contains multiple commands. Both the full accepted-command set and the 432-row strict subset must agree before the result can route future work.

The names in the report are now deliberately unromantic:

• accepted_command;
• last_token_recovery; and
• system_fallback.

None is called direct access to model intent.

Bug three: the same position was not the same policy state

Our expanded A110A census had found 21 legal actions outside the displayed candidate set. A complete follow-up audit showed that all 21 were explicit accepted commands, not loose token recovery. Twenty followed a board-display tool response and one followed a legal-move-list response. Every original candidate-bearing response had requested a tool and embedded an in-set command; the later response revised it after receiving more state information.

This matters because an immediate response to a candidate header and a later response after a board or legal-move tool are different information states. Pooling them could make an information-acquisition effect look like proposal or selection regret.

The amended A110B rows retain attachment_scope:

• immediate_header; or
• same_position_followup.

The accepted-command sample contains 419 follow-ups but only 14 immediate-header rows. Those 14 are distributed 4, 4, 3, and 3 across top-10, top-2, top-3, and top-5.

That is not enough support for a directional mechanism claim. We froze a minimum of 50 rows per required scope and a maximum bootstrap interval width. Because the immediate-header scope fails by construction, this retrospective sample cannot authorize a proposal-versus-selector promotion—even if all of its means point in the same direction.

That may sound like designing an experiment to return “defer.” It is the opposite. We discovered, before seeing engine outcomes, that the sample was not designed to distinguish information state from selection. The honest output is a better next experiment: a prospectively matched information ablation.

Bug four: one engine at two depths is not independent evidence

The historical candidates came from Stockfish. A deeper Stockfish pass can be blind to which move the model selected, but it still shares an evaluation function, search family, and network lineage with the proposer.

We had used “independent evaluator” too casually. The corrected language is “higher-budget, choice-blind Stockfish proxy.” Two depths of the same binary measure budget stability, not evaluator independence.

The amended cache now evaluates every legal root move twice:

• primary_100k: 100,000 requested nodes; and
• confirmation_400k: 400,000 requested nodes.

Each move starts from a cleared hash. Both passes use the same exact binary identity so the depth comparison is interpretable. They are separate searches, not independent scientific replications.

The expanded plan contains 24,915 legal moves. Thirteen terminal successors receive exact zero-node labels, leaving 24,902 nonterminal labels. Across both passes, the exact static request is 12.451 billion nodes.

The confirmation gate does more than compare a final categorical route. It checks:

• best-legal move agreement;
• best-proposal move agreement;
• proposal, selector, and total-regret drift;
• crossings of the 0.02 materiality threshold;
• stability by legal-move-count bucket; and
• per-state Spearman correlation over every legal move.

Even passing those checks would not remove advisor/evaluator coupling. A different evaluator family or network remains necessary before a mechanistic headline.

Weighting is part of the estimand

The 1,000 attempts were sampled with equal quotas by candidate variant and parser-direct versus fallback origin. But source-cell inclusion fractions vary substantially. A unit-weighted sample answers a quota-standardized question, not “what happened on an average frozen corpus attempt.”

The accepted-command split adds another subtlety: equal direct quotas do not guarantee equal accepted-command counts by variant. Calling a unit-weighted accepted subset “variant-balanced” would be false.

The amendment defines three separate estimands:

1. true equal-variant weighting within the analyzed scope;
2. corpus weighting by inverse source-cell inclusion probability; and
3. equal-variant, equal-unique-state weighting so repeated attempts at one state cannot dominate.

The frozen sample contains 947 unique states, including 42 states represented more than once. Sixteen rows occur at a twofold-repetition state and five are threefold-claimable from reconstructed game history. Routing repeats after excluding those positions because a static FEN evaluator does not fully encode claimable-repetition history.

The gate is supposed to be hard to pass

The amended directional decision requires agreement across:

• two evaluator budgets;
• three weighting schemes; and
• six analysis scopes: accepted command, all parser-direct, strict command, non-repetition, immediate header, and same-position follow-up.

That is 36 route calculations, plus the quantitative evaluator-stability gate and the minimum-support rule.

The fallback contrast cannot establish an interface mechanism because fallback status is outcome-defined. It may prioritize a future matched interface study, but it cannot turn into a causal claim.

These requirements are intentionally conservative because A110B is a retrospective diagnostic. Its job is to choose the next prospective experiment, not to manufacture a mechanism from historical traces.

What the refusal bought us

The original plan would have been cheaper and easier to explain. It also could have produced a polished answer to an ill-defined question.

The audit gave us five concrete improvements:

1. the named WDL now matches the measured WDL;
2. parser sources no longer masquerade as one intent variable;
3. information-state follow-ups are visible rather than pooled;
4. repeated states and repetition history enter sensitivity gates; and
5. same-engine depth stability is no longer called independent evaluation.

Most importantly, it established a precedent for the laboratory: a frozen sample is not sacred when a pre-outcome audit finds that its estimand is wrong. The original preregistration remains preserved. Amendment 1 records every change. No engine label existed when it was written. Both old and new hashes remain in the chronology.

Reproducibility boundary

The amended sample and manifest reproduce byte for byte. The live-screen guard has also been tested against the active DeepSeek sessions: it refuses before opening Stockfish or creating a cache.

No A110B regret result exists yet. When the evaluation host becomes idle, the 12.451-billion-node cache can run. Its output will be a retrospective Stockfish-proxy triage result, not proof that Luna is a stronger critic than proposer.

The experiment we refused to run was not wasted work. It was the draft that showed us what the real experiment had to measure.

---

Reproduction

The governing protocol is research/experiments/A110B_AMENDMENT_1.md. The engine-free plan can be rebuilt with:

uv run python -m research.experiments.run_a110b_regret_decomposition --mode plan

The frozen sample-plan JSONL SHA-256 is a27b84f21ac652c3dce06624556db6048d999875bfe44f419a60940d473a975a; the generated sample-manifest file SHA-256 is 25e0e80ec9b46ca41e8cefd0b0b6995e8dd136bfc2279e6569e3ba0b8cf2b362. Plan construction makes zero provider and engine calls.