The Engine That Had to Wait

A label can contaminate an experiment before a model ever sees a prompt. Our two-pass Stockfish stage therefore begins with a refusal—and treats disagreement as exclusion, not truth to be negotiated.

It is easy to describe position labeling as housekeeping. Collect some chess positions, ask a strong engine whether each one is tactical, ambiguous, or quiet, and balance the final benchmark across those categories.

But that innocent sequence hides three decisions that can manufacture a result. Which positions were allowed into the pool? Which engine observation selected them? And what happens when the label changes on a repeated search?

For E020 and E021, those decisions matter because the experiment is designed to separate move generation from move selection. The candidate list shown to a model is part of the treatment. The difficulty class is part of the sampling design. The eventual move-quality score is the outcome. Letting one of those objects leak into another would make a precise-looking experiment answer a different question.

So we built the labeling stage to wait.

The executable runner exists. Its schemas, cache identities, stability rules, and refusal paths are tested. But the real Stockfish stage has not run. It may not open the engine until the live replication cohort is terminal, content-addressed, and jointly cleared against the candidate pool. No model or provider outcome has been collected for E020 or E021. Nothing in this chapter changes A105's separate paid-execution gate.

That absence is not delay disguised as rigor. It is the method.

Why a live cohort makes a frozen pool provisional

The paired-position benchmark must not contain positions from either full-game pilot cohort. Otherwise a “new” diagnostic case could be a state the same model already generated or encountered in the experiment that motivated it.

Luna's pilot is frozen. DeepSeek's replication is not yet a terminal evidence object. While that cohort remains live, a new completed game can add new board states. A pool found disjoint from yesterday's partial cohort can overlap tomorrow's terminal cohort without changing a single byte in the pool itself.

This is why a point-in-time overlap check cannot authorize labeling. The gate requires three content-addressed objects:

1. the exact frozen overcomplete position pool;
2. the terminal Luna cohort; and
3. the terminal DeepSeek cohort.

The qualification is then rebuilt from those exact inputs. It checks both normalized positions and color-mirrored families. The persisted qualification must equal the fresh reconstruction byte for byte, and every source hash must still match. If the terminal DeepSeek freeze is absent, incomplete, stale, or different, the runner stops before constructing an engine process.

This ordering closes a temporal leakage path. It prevents us from spending compute on labels that would later become ineligible, but efficiency is not the main point. The stronger guarantee is that benchmark membership cannot be defined against a moving reference set.

The pool is frozen before the engine knows its classes

There is a second protection upstream. The overcomplete pool is built engine-blind. It freezes realistic positions with their provenance, source-game lineage, normalized identity, mirror family, phase, side to move, and partition-isolation keys before Stockfish supplies a tacticality label.

That pool must already satisfy legal-width and reserve constraints. Every position needs at least ten legal root moves because top-10 is an actual experimental condition, not a label pasted onto a narrower action set. Every split, phase, and color cell needs enough surplus to survive later exclusions.

We do not know in advance whether the engine will call a position tactical, ambiguous, quiet, or unclassified. That ignorance is deliberate. If we searched for positions after observing the desired class or a promising outcome value, the selection process would become an invisible optimization loop.

The reserve can still fail. A class quota may remain short after all registered gates. In that case the final freeze stops. The remedy is not to change a threshold, borrow a convenient position from another split, or lower the stability standard after inspecting the shortfall.

Two passes, one narrow question

Each eligible pool position is scheduled for two Stockfish 18 searches. Both use MultiPV 10, 50,000 nodes, one thread, a 256 MB hash, WDL reporting, and the same registered options. Clear Hash is sent before every uncached position in every pass. The binary hash, engine identity, options, node budget, FEN, role, and pass number are all part of the immutable cache key.

Pass A and pass B are not independent replications. They are two cleared-hash observations from the same engine family under the same fixed-node contract. Their purpose is narrower: identify labels and candidate orderings that are stable enough to define the benchmark.

A case is eligible only if both passes return ten unique legal root moves and all of the following hold:

• the top move is unchanged;
• ranks one through six are identical;
• the two top-10 sets have Jaccard overlap of at least 0.80;
• drift in the best-versus-second WDL gap is at most 0.02; and
• both passes assign the same registered class.

The class rules use root-player WDL expectation. A winning mate or a best-versus-second gap of at least 0.20 is tactical. At least three of the top five moves within 0.03 of the best, without a winning mate first, is ambiguous. A nonforcing best move with a gap strictly between 0.03 and 0.20 is quiet. Everything else is unclassified.

These are operational strata, not claims about a human essence called “tacticality.” The names make the design readable; the thresholds define the actual sampled population.

Disagreement is not a puzzle for a third pass

Suppose pass A calls a position tactical and pass B calls it ambiguous. Or the same top move survives while ranks two through six reorder. It is tempting to run a third search, increase the node budget, take a majority vote, or keep the pass whose answer makes the quotas work.

Every one of those responses changes the registered estimand.

A third pass creates an adaptive stopping rule: only unstable cases receive extra evidence. A larger search answers a different fixed-node question. A majority vote makes stability depend on an unregistered number of repeats. And quota-aware adjudication allows the desired dataset composition to choose the label.

Our rule is simpler: disagreement becomes exclusion.

The runner records why the case failed, but it does not publish a class for that case. It never substitutes pass B's list, averages the two orderings, or silently promotes a near-threshold position. Instability is treated as failure to meet an inclusion criterion, not as noisy evidence for one side of a binary truth.

This rule has a cost. The final benchmark describes positions with stable fixed-node Stockfish strata, not all chess positions. Exclusion can change the difficulty distribution and may disproportionately remove volatile tactical states. That is why the exclusions and their reasons belong in the eventual flow report. A clean label is not a license to forget which cases were too unstable to label.

The benefit is interpretability. Readers can reproduce the inclusion rule without reconstructing an analyst's judgment after the fact.

Pass A presents; pass B only audits

For every stable case, the candidate list used by the experiment is exactly pass A's ordered ranks one through ten. Each item retains its UCI and SAN move, root-perspective centipawn-or-mate score, WDL expectation, search statistics, and complete legal principal variation. The canonical JSON receives its own hash, and the final manifest hashes the ordered mapping from position keys to candidate-list hashes.

Pass B never replaces that list—even when its list looks more attractive.

This asymmetry is important. The top-2, top-5, top-10, and best-omitted conditions must all be deterministic slices of one frozen action universe. If we chose between pass A and pass B after comparing them, the treatment itself would inherit a hidden selection step. Stability qualifies pass A; it does not optimize it.

The rendered model prompts later remove engine scores, ranks, and source-order information as registered. But removing metadata at presentation time is not enough. The underlying list identity must already be fixed.

The labeling engine cannot grade its own exam

Stockfish's role in this stage is proposer and classifier. Its values may decide which stable stratum a position occupies and which candidate moves define the treatment. They are not the primary outcome evaluation.

After the final 600 positions are selected and frozen, a different engine family and binary must score the model's choices for the primary regret endpoint. Reusing Stockfish is allowed only as a separately named, choice-blind internal-proxy sensitivity.

This separation blocks a circular success criterion. If the engine that proposed the candidates, ranked them, selected the cases, and defined their classes also supplied the primary outcome, the experiment could measure agreement with one engine's policy while presenting the result as general move quality.

A different-family evaluator does not create ground truth. It creates a more honest disagreement surface and permits the narrower claim that a result is not merely proposer self-agreement. Without that evaluator, the evaluator- independent E020/E021 headline remains unavailable.

A resumable process that cannot publish half a result

Fixed-node labeling is expensive enough that a crash should not erase verified work. Each pass therefore writes an immutable, content-addressed cache record. On restart, the runner revalidates the schema, engine identity, FEN, pass number, candidate legality, complete PVs, and payload hash before reusing it.

Resumability does not weaken the evidence boundary. Partial caches are diagnostic artifacts, not proof that the label stage completed. The complete stage directory becomes visible only after every planned pass has been accounted for as either a stable label or a registered exclusion, every cache index verifies, the engine binary still hashes identically, and the host is idle again.

The same idle-host gate runs before engine startup, before every uncached search, and before publication. This keeps the CPU-heavy label stage from competing with live evaluation suites whose runtime and behavior are themselves under measurement.

What exists, and what does not

Today we have an executable refusal architecture:

• a terminal-cohort and joint-leakage authorization gate;
• a frozen engine-blind pool contract;
• a two-pass fixed-node runner with immutable caches;
• exact candidate-list and lineage hashes;
• registered stability, classification, and exclusion rules; and
• a final manifest that cannot represent a partial run as complete.

We do not yet have real two-pass labels, a final 600-position freeze, a different-family evaluator result, any E020/E021 model response, or any causal finding from those experiments. The DeepSeek cohort is not declared terminal here. No provider was contacted by the label runner. A105's paid execution remains unauthorized and separate.

The next valid event is not “Stockfish produced some labels.” It is that every upstream evidence object became terminal, matched its recorded identity, and authorized the engine to begin.

Until then, the most scientifically meaningful output of the runner is a refusal.

What would change our mind?

We would revise this protocol before outcomes only if an implementation audit found that the registered two-pass checks do not measure the intended fixed- node stability, that the classification rules are internally inconsistent, or that the exclusion mechanism cannot be reproduced from immutable inputs.

After outcomes exist, instability itself may motivate a separately preregistered study: for example, whether candidate-rank volatility predicts model errors. That would use the excluded cases as a new target population. It would not retroactively admit them into E020/E021 or reinterpret disagreement as a label.

The discipline is the point. A benchmark is not just a table of positions. It is the history of every decision that allowed those positions to become data.

Reproduce the fail-closed boundary

The reference implementation is research/experiments/run_e020_e021_label_stage.py. Its preflight mode checks terminal qualification and pool eligibility without opening Stockfish. Its label mode adds the engine-identity, idle-host, two-pass cache, and complete- publication gates.

The current real label stage remains unexecuted and unauthorized while terminal E001 qualification is absent. That is the expected result—not a missing one.