Full Games, Not Chess Puzzles

Why a move benchmark and a playing agent answer different questions

A chess puzzle begins at the moment the puzzle setter finds interesting. The pieces are already arranged. The tactic exists. The task ends as soon as the right move is named.

A chess game is less polite. The agent has to create the position it will later be judged in. It must remember what happened, survive its own inaccuracies, recognize when the character of the position changes, and keep using its tools correctly after dozens of decisions.

That distinction matters for language models because a model can look competent at isolated move choice and still be a fragile player. It can also lose complete games despite possessing useful local chess judgment. A puzzle measures a decision. A game measures a decision-making system moving through time.

The path is part of the problem

Suppose two agents reach the same FEN. One sees only the current board. The other carries a long transcript containing tool calls, rejected actions, analysis of plans that no longer apply, and descriptions of pieces that have since disappeared.

They face the same legal position but not the same inference problem.

The second agent must recover the signal from its own history. Earlier analysis can serve as memory, but it can also anchor the model to a stale plan. The transcript can preserve intention while consuming the attention needed to calculate. A chess engine treats the board as state. A conversational agent may treat the entire path as state, even when much of that path is now misleading.

Complete games therefore test at least six things that a frozen puzzle does not:

1. State continuity: Does the agent maintain an accurate board across turns?
2. Error recovery: Can it adapt after choosing an inferior move?
3. Plan revision: Does it abandon plans invalidated by the opponent?
4. Phase transition: Does its reasoning change from opening to middlegame to endgame?
5. Protocol endurance: Does it continue emitting legal, correctly formatted actions?
6. Resource accumulation: How do tokens, latency, and tool turns scale with game length?

Those are not peripheral engineering details. They are components of the player.

What 220 Luna games actually contain

The completed Luna pilot contains 220 clean full games across 22 adaptive ladder blocks. Of those games:

• 166 ended in checkmate;
• 45 reached the 200-ply administrative limit;
• five ended with insufficient material; and
• four ended by fivefold repetition.

That means roughly one game in five was declared drawn by the experimental cutoff rather than by a game-theoretic chess result. This is not a reason to discard the games. It is a reason to model the stopping rule as part of the protocol.

The trajectories also differ substantially in length. Descriptively, across all tested Elo anchors for each condition, median game length was:

Condition	Clean games	Median plies	Median total tokens
Regular	10	167.5	326,245
Top 10	10	81.5	95,570
Top 5	60	127.5	127,662
Top 3	80	123.0	101,295
Top 2	60	89.0	59,563

These rows are not a causal comparison. The ladder deliberately moved stronger conditions to stronger opponents, so variant, opponent Elo, and sampled game trajectory are entangled. The table instead demonstrates why cost and strength cannot be inferred from a single per-move benchmark. The unit of deployment is the trajectory.

Full games expose compounding error

If an agent has a small independent probability of making a serious error on each move, survival falls rapidly with game length. But chess errors are not independent. One inaccuracy can create a harder position, which increases the chance of another error. A mistaken plan can persist across several turns. A hallucinated board fact can contaminate every calculation downstream.

This creates an error cascade:

small inaccuracy
      ↓
harder or unfamiliar position
      ↓
more candidate ambiguity
      ↓
shallower or less grounded calculation
      ↓
decisive blunder

A puzzle dataset usually samples the final arrow. A full game reveals the system that produced it.

That is why “first decisive blunder” is more informative than final result alone. We want to know the ply where expected outcome first collapses, whether the position was already deteriorating, whether the best move appeared in the candidate set, and whether the model selected it. The resulting hazard curve can distinguish sudden tactical failure from gradual strategic drift.

But full games are terrible causal instruments

The strength of full games is also their central weakness. After two conditions choose different moves, they no longer face the same positions. A top-two agent may reach simpler endgames while an unaided agent enters tactical chaos. Later move quality then compares both decision ability and the state distributions created by earlier decisions.

Four problems follow.

Trajectory confounding

The treatment changes the future test set. Better early moves can make later moves easier—or produce longer resistance against a stronger opponent.

Opening funnels

Starting every game from the initial position does not create ten independent chess worlds. Deterministic or low-variance opponents can repeatedly steer games through related openings.

Survivor bias

Only agents that survive reach late endgames. Comparing late-game accuracy can make a weaker system look stronger because its difficult games ended earlier.

Administrative outcomes

A fixed ply cap may call a technically won position a draw. Changing the cap, adjudication engine, or tablebase policy changes measured score without changing any move the model made.

Full games tell us whether a system works end to end. They do not, by themselves, tell us why.

The laboratory needs two instruments

Our research program therefore uses a two-level design.

Instrument 1: paired positions

Every condition faces exactly the same frozen positions. Candidate sets are cached and nested. Presentation order is repeated. We measure selected move, WDL regret, catastrophic blunders, legality, latency, and tokens.

When the intervention is the only controlled difference, this instrument is designed to estimate local causal effects more precisely than divergent full games. The registered study has not yet produced those estimates.

Instrument 2: complete games

Interventions that survive the paired benchmark are promoted to balanced opening mini-matches with both colors. We measure score, failure hazard, trajectory cost, state drift, and system reliability.

This instrument tests whether a local improvement compounds into a useful player.

Neither replaces the other. A paired benchmark without games can optimize a move selector that cannot sustain a match. Games without paired positions can produce an exciting leaderboard with no explanation.

The context experiment

One of the cleanest tests reuses the same positions under different histories. For each frozen FEN, ask the model to move under four conditions:

1. fresh context containing only the current position;
2. the exact full-game transcript that produced it;
3. a compact reconstructed state plus the last few moves; and
4. a deliberately irrelevant but length-matched chess transcript.

If full history helps, the model is using trajectory information not present in the board. If compact reconstruction wins, the bottleneck is context hygiene. If an irrelevant transcript hurts equally, attention load is sufficient to explain the degradation. If identical FENs reached through different histories produce different moves, path dependence has become measurable.

The paired-run event-ledger implementation is designed to record the complete prompt, response, board, candidate order, selected move, tokens, latency, and configuration fingerprint. Once observations are collected, those fields will permit the test without reconstructing history after the fact.

What would change our mind?

The full-game emphasis would weaken if paired-position quality predicted match strength almost perfectly across models and scaffolds. It would also weaken if fresh-context and full-history decisions were indistinguishable at every game phase, or if first-blunder hazards were constant once local position difficulty was controlled.

Conversely, a growing fresh-versus-history gap by ply would show that long-horizon agent behavior is a distinct capability. So would interventions that leave single-position accuracy unchanged but materially improve game survival.

The actual object of study

The question is not whether puzzles or games are “more realistic.” They measure different objects.

A puzzle asks whether the model can solve the position placed in front of it. A game asks whether the entire system can keep placing itself in positions it can solve.

For agents, intelligence is not only the quality of one answer. It is the shape of the trajectory created by many answers, each conditioned on the consequences of the last.

---

Reproducibility note

The counts and descriptive table in this post are generated by research/analysis/game_trajectory_summary.py from the content-addressed Luna freeze. They include clean game records only and intentionally pool adaptive anchors only for trajectory description—not causal condition comparison. The next analysis release will add per-ply engine evaluation and first-decisive- blunder curves after the live DeepSeek suites no longer share the host.