Where Does the Game Fall Apart?

Elo hides the difference between a bad move, a broken interface, and a failed trajectory

Two agents lose to the same opponent.

The first plays legal chess for sixty moves and misses a tactical resource. The second repeatedly calls the wrong action until the harness declares a loss. A leaderboard records both as zero points.

For deployment, that may be appropriate: neither system completed the task. For research, treating them as the same failure throws away the explanation.

Aggregate Elo answers whether the whole protocol wins. It does not tell us where the protocol breaks.

The first DeepSeek anchor makes this concrete

At the provisional 2026-08-04T21:15:58Z trace snapshot, DeepSeek V4 Flash regular play had completed ten clean games against Stockfish 1320 with a 0–2–8 record. The terminal reasons were:

• four checkmates;
• four “too many wrong actions” forfeits;
• one 200-ply administrative draw; and
• one insufficient-material draw.

Half of the losses came from the interface budget rather than checkmate.

Across the 43 clean DeepSeek games available in that snapshot, ten ended through wrong-action exhaustion. None ended through a provider error. These facts matter independently:

• provider reliability was good;
• agent–tool protocol reliability was not uniformly good;
• chess strength among legally completed trajectories is a separate question; and
• system strength properly includes both.

The Luna pilot has a different profile. Across 220 frozen clean games, none ended through wrong-action exhaustion, 166 ended in checkmate, and 45 hit the 200-ply administrative cutoff.

Similar ratings can conceal different failure mixtures. Different ratings can also reflect interface behavior rather than chess judgment alone.

A failure stack

We organize failures into layers.

Layer 0: infrastructure

Did the provider return a usable response? Were retries exhausted? Did the process, network, or serializer fail?

Infrastructure errors are not chess decisions. They remain reliability evidence but are excluded from move-quality estimates and restored when a paired cell is missing.

Layer 1: interface

Did the model follow the action protocol? Could the response be parsed? Was the selected move legal? Did it exhaust wrong-action attempts?

Interface failures count against end-to-end system performance. In diagnostic move-quality analysis they receive maximal regret and a separate rate; silently dropping them would reward the least reliable model.

Layer 2: local chess decision

Given a legal move, how much expected outcome did it lose relative to the best available move? Was a good move present in the candidate set? Did the model choose it?

This layer produces WDL regret, centipawn regret, best-move rate, policy coverage, and catastrophic-blunder indicators.

Layer 3: mechanism

Why did the local decision fail? Candidate omission, comparison, state grounding, search horizon, value misranking, stale plans, and context degradation are mechanistic labels.

They cannot be inferred reliably from prose alone. Each requires a controlled contrast.

Layer 4: trajectory

When did failure emerge, how did it change later position difficulty, and could the agent recover? This is where first-blunder hazard, survival, phase effects, and history dependence enter.

The same game can contain events at several layers. A provider retry may recover, a legal inaccuracy may worsen the state, and a later tactical blunder may end the game. The event ledger preserves the sequence rather than forcing one global label.

Observable codes before causal stories

The project now has a machine-readable failure taxonomy. It distinguishes direct observations from hypotheses.

PROTOCOL_WRONG_ACTION is observable: the registered attempt budget was exhausted. CATASTROPHIC_BLUNDER is operational: WDL regret exceeded 0.20. ADMINISTRATIVE_CENSOR is defined by the stopping rule.

“Weak board grounding” is not directly observable from a loss. It becomes a hypothesis supported only if equivalent board representations or fresh-state reconstruction repair the decision. “Search horizon” requires an intervention that supplies replies or principal variations. “Value misranking” requires fully specified successor states.

This discipline prevents explanations from being retrofitted to colorful model analysis.

Finding the first decisive event

Final checkmate can occur many moves after the game is effectively lost. To locate the collapse, we evaluate every model move with a fixed-node analysis engine.

For a model move at ply t:

regret(t) = best expected WDL before move
            − selected-move expected WDL before move

A move is provisionally catastrophic when regret is at least 0.20 expected WDL. The first such move defines the game's first-catastrophic ply. Games without one are censored at their final evaluated model move rather than treated as failures at infinity.

The analysis produces:

• first-failure hazard in 20-ply bins;
• survival without catastrophic error;
• opening, middlegame, and endgame event counts;
• mean and median move regret;
• position difficulty before the event;
• candidate inclusion and selection at the event; and
• termination reason after the event.

This distinguishes sudden tactical collapse from gradual accumulation.

Why hazard is better than average accuracy

Suppose one system plays ninety excellent moves and then blunders once. Another loses small amounts of WDL every ten moves but never crosses the catastrophic threshold. Their mean move accuracy and match outcomes can disagree.

A hazard curve asks a different question: conditional on surviving without a decisive failure so far, how likely is the first failure now?

The curve can reveal opening grounding failures, middlegame branching overload, endgame context problems, growth with transcript length, and scaffold effects that delay rather than eliminate collapse.

Late-game hazard must be interpreted with survivor bias. Only games that avoid earlier failure reach later bins. We therefore report at-risk counts beside every hazard estimate.

The annotation pipeline is deliberately paused

Per-ply engine annotation is CPU intensive. The current opponent and candidate engines use wall-clock limits, and four DeepSeek suites still share the host. Running deep analysis now could change their realized search and contaminate the experiment.

The annotator therefore refuses to run while any Luna or DeepSeek evaluation screen exists. Once the host is idle, it will:

1. verify the content-addressed Luna source freeze;
2. pin and hash the analysis-engine binary;
3. evaluate the best move and played move at fixed nodes;
4. save one resumable annotation file per game;
5. preserve source-game paths and hashes; and
6. generate failure-hazard summaries from those annotations.

The implementation and unit tests exist now. The result does not.

This is what experimental patience looks like: infrastructure can advance while analysis waits for a non-confounded machine state.

Candidate assistance can repair different layers

A shortlist may reduce failure through several paths.

• Fewer illegal or malformed choices improve interface reliability.
• Better policy coverage reduces local chess regret.
• Lower branching factor reduces search failures.
• Recognizable move notation refreshes state grounding.
• Shorter games or cleaner positions change trajectory exposure.

If top-2 merely prevents wrong-action forfeits, its Elo gain is primarily an interface effect. If protocol rates are unchanged but WDL regret falls, the gain is chess quality. If early move quality is similar but failure hazard moves later, the scaffold improves trajectory stability.

The reports must show all three.

The paired study adds local resolution

The 8,800-cell paired schedule will produce condition summaries for legal-action rate, provider errors, effective WDL regret, catastrophic blunders, best-move selection, latency, tokens, and disagreement across the two total presentation orders used on the 200 ambiguous positions.

Primary contrasts compare oracle top-2 with naked play and oracle top-10 with top-2 on identical positions. Confidence intervals resample positions, not individual order permutations.

This local evidence connects candidate assistance to the failure stack. Full games then test whether the repaired layer changes survival.

History as a causal variable

Failure hazard may rise with ply because positions become harder. It may also rise because the transcript becomes longer.

To separate them, the same FEN can be evaluated under fresh context, exact game history, compact reconstructed state plus recent moves, and irrelevant length-matched chess history.

If exact history hurts but compact state repairs the move, context maintenance is the bottleneck. If every condition struggles equally, position difficulty is sufficient. If irrelevant history hurts, attention load matters independently of chess content.

The unit of comparison is the same board, not two unrelated late-game positions.

Administrative draws need three treatments

The Luna pilot recorded 45 games as draws at 200 plies. The protocol score should retain that rule because it governed the experiment. Chess interpretation needs two additional views: independent fixed-node adjudication of the final position, and censoring sensitivity that removes administrative outcomes.

If candidate rankings change across those treatments, the stopping rule is material. Reporting only the most favorable treatment would be another form of post-hoc protocol selection.

What would change our mind?

The trajectory hypothesis weakens if first-blunder hazard is constant across ply after controlling position difficulty, or if fresh-context and full-history decisions are indistinguishable. It strengthens if compact state reconstruction repairs late-game decisions without changing the board.

The interface explanation for DeepSeek weakens if wrong-action rates disappear under a frozen one-move parser while chess regret remains high. It strengthens if legal-action compilation closes much of the system-score gap.

The candidate scaffold is not a chess-quality intervention if its apparent gain vanishes after interface forfeits and administrative draws are separated. That is a possible result, and the analysis is designed to expose it.

What a useful leaderboard should show

A single rating can remain the summary, but it should open into a failure profile:

system score
├── provider reliability
├── protocol and legality
├── local move regret
├── candidate coverage and selection
├── first-failure hazard
├── phase and context sensitivity
└── cost and latency

Two systems with the same Elo may require completely different improvements. One needs a legal-action compiler. One needs a policy. One needs deeper reply search. One needs compact state. One needs a calibrated value function.

“Where does the game fall apart?” is the question that turns a leaderboard into an engineering program.

---

Reproducibility note

Current terminal-reason counts come from experiment_tracking/games.csv and the content-addressed Luna cohort. Per-ply results are not yet claimed. research/analysis/trajectory_annotation.py and research/analysis/failure_hazard.py implement the guarded fixed-node pipeline; research/schemas/failure_taxonomy.yaml defines observable versus mechanistic codes. The trajectory release will name its source freeze, analysis-engine hash, node budget, and censoring rules.