Is the Model a Searcher, a Critic, or a Policy?

Candidate assistance reveals roles that aggregate Elo hides

A chess move arrives as one action, but producing it requires several different computations.

The agent must represent the board, propose legal moves, imagine replies, evaluate resulting positions, and decide when its analysis is sufficient. A single neural network may approximate all of those operations, yet success or failure does not tell us which operation was responsible.

Our Luna candidate experiment makes that ambiguity impossible to ignore.

Unaided, the system's joint local rating is estimated near 1067. With two randomized, unranked Stockfish proposals, it rises near 2832. The model did not receive the engine's score or rank. It still had to understand the position, compare the moves, and commit.

The result is consistent with a model that is a much better critic than proposer. It is not yet proof of one.

Five roles inside one move

A useful decomposition contains at least five roles.

State estimator

Constructs an accurate representation of the current board, side to move, castling rights, tactical constraints, and relevant history.

Policy or proposer

Assigns probability mass to plausible legal moves. A strong policy ensures that the consideration set contains useful actions.

Transition model or searcher

Predicts how candidate moves change the board and which replies matter. Search may be explicit tree expansion or implicit internal calculation.

Value function or critic

Ranks successor positions by expected outcome. It must compare material, initiative, king safety, structure, tactics, and long-term prospects.

Controller

Allocates computation: how many candidates to examine, which branches to deepen, when to call a tool, when to stop, and how to turn analysis into a valid action.

In a normal prompt, all five sit behind one completion. In a full game, their errors compound into one W-D-L result. Candidate scaffolding is valuable because it cuts into the pipeline between policy and search.

What top-2 assistance removes

The Stockfish shortlist repairs several problems at once:

• it guarantees legal proposals;
• it raises the chance that a strong move enters consideration;
• it reduces the search branching factor;
• it gives the model recognizable chess notation; and
• it signals that the listed moves deserve attention.

What remains is also substantial:

• the board still has to be interpreted;
• candidate consequences still have to be calculated;
• the engine-best candidate is not labeled;
• close moves still require comparison;
• the model can reject the list; and
• the final action must still be emitted correctly.

A dramatic top-2 gain therefore shows that the unaided bottleneck lies somewhere inside the repaired bundle. It does not isolate proposal recall from branch control, authority signaling, or attention.

Behavioral signatures of different bottlenecks

Each role predicts a different pattern across controlled interventions.

Observation	Most compatible bottleneck
Best move absent from self-generated shortlist	Policy generation
Best move present but rejected	Critic or comparative search
Successor boards repair selection	Transition/state update
Opponent best reply repairs selection	Search horizon
Correct ranking but illegal final action	Controller/interface
Strong fresh-context play, weak long-history play	State/context maintenance
Values discriminate winners but moves remain weak	Policy/search rather than critic
Good candidates plus deep PV still misranked	Value function

This is the core research opportunity: turn vague “reasoning ability” into a measurable capability profile.

The proposer–selector cross

The cleanest experiment crosses who proposes with who selects.

Proposer	Selector	What it measures
Model	Model	End-to-end unaided decision
Model	Engine	Quality and recall of the model's policy support
Engine	Model	Model selection, search, and value over strong proposals
Engine	Engine	Candidate-set ceiling and harness calibration
Random legal	Model	Ability to reject low-quality distractors

Suppose model-propose/engine-select remains weak while engine-propose/model-select is strong. The policy is the primary bottleneck. If the reverse occurs, the model can generate ideas it cannot evaluate reliably. If both are weak but engine-engine is strong, the remaining state/search/value stack is inadequate.

The cross should use identical positions and preserve every proposed move. A model's final selection cannot reveal whether a strong rejected idea was ever considered unless the shortlist itself is an experimental output.

Measuring policy quality

Policy quality is not simply “did the model name Stockfish's first move?” Chess often has several acceptable actions. We measure:

• engine-best recall in the first k proposals;
• best available source rank;
• WDL coverage of the shortlist;
• diversity across strategic move families;
• legality and duplicate rate;
• sensitivity to notation and board representation; and
• recall as a function of tactical motif and game phase.

A model can have low top-1 accuracy and still provide a useful policy if its top-five support reliably contains a near-optimal move. Conversely, fluent lists of five plausible moves may have poor coverage if all share the same tactical blind spot.

The candidate-count paradox adds another constraint: policy breadth is useful only if the downstream critic can process it.

Measuring the critic

Critic tests should minimize the need to generate or search.

One method presents two fully specified successor positions and asks which is better for the side to move. Another shows a candidate plus the opponent's best reply. A stronger version supplies a short principal variation and asks for an outcome probability.

Critic quality includes:

• pairwise ranking accuracy;
• calibration of win/draw/loss probabilities;
• regret of the selected successor;
• consistency under color reversal and equivalent notation;
• transitivity across three-way comparisons; and
• resistance to superficial material or authority cues.

A critic that chooses the right move for the wrong reason may fail under a slightly deeper reply. Calibration and adversarial contrasts matter more than agreement on easy positions.

Measuring search

Search sits between proposal and value. To isolate it, vary the amount of transition information externalized while holding candidate moves fixed:

1. move notation only;
2. successor FEN after each move;
3. successor FEN plus opponent best reply;
4. a short principal variation; and
5. principal variation plus hidden-from-model evaluation for analysis only.

If successor boards produce a large repair, the model struggles to update the state internally. If opponent replies matter, it struggles to identify tactical responses. If short PVs help but values are still misranked, the critic is the remaining bottleneck.

Search depth can also be inferred from adversarial candidate sets: delayed poison, quiet defensive resources, horizon tactics, and moves whose refutation requires one additional ply beyond the obvious line.

The controller is not bookkeeping

DeepSeek V4 Flash makes the controller problem visible. At the provisional 2026-08-04T20:50Z snapshot, 42 clean games had consumed 22.74 million recorded tokens and 155.96 aggregate evaluation worker-hours. More than 91% were completion-side tokens, including provider-reported reasoning usage.

The issue is not merely price. A system that cannot allocate or terminate search may be too slow even when its tokens are inexpensive. Meanwhile, Luna top-2 was both the strongest scaffold and descriptively the lowest-token Luna variant per game across its tested anchors.

A capable controller should widen search only when uncertainty justifies it, reuse state instead of redescribing it, stop refuted branches early, and expose confidence so external tools can intervene selectively.

The executable diagnostic study

The paired benchmark now has more than a conceptual outline.

The registered 600-position design expands into 8,800 planned model calls:

• 600 positions × two models × six core conditions = 7,200 cells;
• four candidate-bearing conditions receive one second presentation order on each of 200 ambiguous positions for both models = 1,600 additional cells.

Thus v1 has two total orders for those cells, not four repeated permutations. The broader roadmap's four-or-more-order design is a separately preregistered order-focused experiment or a versioned v2 extension; it is not silently part of the locked 8,800-cell schedule.

Every cell binds:

• position and candidate-cache hashes;
• model and condition;
• exact candidate source ranks;
• presentation seed and order permutation;
• prompt-template hash;
• tacticality, phase, and side-to-move strata; and
• a deterministic randomization key that interleaves collection over provider time.

The event ledger will retain full prompts and responses when the registered cells are collected. Request telemetry will retain latency, provider-returned model identity, tokens, retries, request IDs, and errors. Illegal moves remain model failures; provider failures are the only cells eligible for restoration.

That distinction prevents reliability problems from being cleaned away as “bad data.”

From diagnosis to architecture

If the proposer–critic interpretation survives, an efficient chess-native agent should not ask one conversational model to improvise every role in prose.

It could use:

1. Deterministic state: a symbolic board and legal-action mask.
2. Compact policy: a small proposal module returning calibrated move mass.
3. Adaptive support: narrow candidates when policy confidence is high; widen only when coverage is uncertain.
4. Branch executor: explicit successor states and independent reply search.
5. Comparative critic: pairwise or listwise WDL evaluation over successors.
6. Uncertainty controller: deeper search or tool escalation near decision boundaries.
7. Action compiler: guaranteed legal UCI emission separated from analysis.

The language model can remain central while no longer carrying every mechanical responsibility in its context window.

Distillation targets

External scaffolds can also become training data.

• Engine proposals supervise a compact policy adapter.
• Model choices over nested sets reveal preference errors.
• Successor-state comparisons supervise a value head.
• Refutation traces supervise branch selection and search depth.
• Confidence versus regret calibrates uncertainty.
• Cases where a wider set hurts supervise the controller's support-size policy.

The goal is not to imitate Stockfish move for move. It is to internalize the specific operation the model currently needs an external scaffold to perform.

This produces a staged research strategy: externalize a capability, measure the repair, identify the minimal sufficient information, then distill or architect that operation into the agent.

What would change our mind?

The “strong critic, weak proposer” interpretation fails if Luna's self-generated shortlists already contain near-optimal moves but the model rejects them. It also fails if engine proposals do not improve held-out paired positions once candidate sets are truly nested.

A search rather than policy explanation becomes stronger if successor states or opponent replies account for most of the oracle-candidate gain. A value bottleneck becomes stronger if the model misranks explicit, deeply analyzed successors. A state bottleneck becomes stronger if equivalent FEN and board renderings cause large inconsistent changes.

The point of the decomposition is that it can be wrong in informative ways.

Beyond chess

The same roles appear in coding, theorem proving, planning, and tool-using agents.

A coding model may generate weak patches but review strong patches well. A proof model may recognize a valid step without proposing it. A planner may evaluate finished plans better than it expands the search tree. A research agent may collect good hypotheses and still compare them poorly when the list grows.

Chess gives us legal actions, exact transitions, strong evaluators, and complete trajectories. That makes the role boundaries unusually observable.

The larger question is not whether an LLM “can play chess.” It is which parts of an intelligent decision system already exist inside the model, which need tools, and which should become separate learned modules.

The top-2 result is valuable because it makes that division of labor visible.

---

Reproducibility note

The policy–critic claims in this post remain hypotheses. The system-level Luna curve is frozen and jointly analyzed; the causal role decomposition awaits E020, E021, and E022. The executable schedule generator, prompt hashes, deterministic candidate cache, request telemetry, event ledger, and reconciliation rules are implemented under research/instrumentation/ and tested before paid diagnostic collection.