Ideas or Vision? Why the Score Alone Cannot Tell

Two chess systems can fail equally often for opposite reasons

Suppose two agents find Stockfish's preferred move 31.5% of the time.

The first agent rarely proposes that move. It includes the move in only 35% of its shortlists—but when the move is present, it selects it 90% of the time. The second agent usually proposes the move. It includes it 90% of the time—but selects it only 35% of the time.

0.35 × 0.90 = 0.315
0.90 × 0.35 = 0.315

Their headline accuracy is identical. Their engineering diagnoses are almost opposites.

The first system is ideas-limited in an operational sense: a strong action often fails to enter its consideration set. The second is vision-limited in an equally operational sense: a strong action is available, yet the system often fails to distinguish it from alternatives. One needs better proposal recall. The other may need a sharper critic, less comparison load, more stable presentation, or explicit state transitions and reply search.

Those labels are not claims about human cognition. “Vision” here does not mean that a language model sees a chessboard as a person does. It is shorthand for observable success at choosing among already available actions. Even that shorthand is provisional until further interventions separate comparison, state transition, search, and value estimation.

This is the central measurement problem behind our next paired-position experiments. Full-game Elo can tell us which complete system won more often. Candidate assistance can tell us that changing the interface helped. Neither, by itself, identifies why.

Figure — E023 observational alias and order-control design

Figure: outcome-blind synthetic design evidence. The diagram contains no Luna or DeepSeek mechanism outcomes; it shows why the next intervention is identifiable under its declared inputs.

One outcome, two hidden stages

A useful first approximation treats move choice as a two-stage process:

1. Generation: does the candidate set contain a sufficiently strong move?
2. Selection: conditional on that move being available, does the model choose it?

Let C be best-move coverage and S be conditional best-move selection. Then the observed best-move rate is

P(best move selected) = C × S

Observing only the product does not identify its factors. This is observational aliasing: different latent mechanisms produce the same measured outcome.

The problem is worse in full games. A loss may begin with a missed candidate, a mistaken comparison, an inaccurate imagined successor, a missed opponent reply, a stale board representation, or a later error caused by an earlier position. Adaptive opponent ratings, different openings, and unpaired game trajectories add still more variation. Elo intentionally compresses all of that into one systems-level number.

Compression is valuable for a leaderboard. It is inadequate for mechanism.

Even a top-2 versus top-10 comparison remains aliased. The larger list changes at least three things simultaneously:

• the number of comparisons;
• the quality and ambiguity of the additional candidates; and
• the realized positions and downstream game trajectories.

If top-10 performs worse, we have learned that the complete top-10 intervention was worse. We have not yet learned that the model was overwhelmed merely by counting to ten.

The paired intervention ladder

The prospective E020/E021 design attacks the first layer of aliasing by showing the same frozen positions under different information conditions. Its core arms include unaided play, self-generated shortlists, nested oracle lists, and a best-move-omitted control.

That creates diagnostically different observations:

• Self-shortlist coverage asks whether the model can generate a strong candidate.
• Oracle-list selection asks whether it can choose that candidate when an external proposer guarantees availability.
• Best-move omission measures behavior when the anchor action is genuinely unavailable rather than merely ignored.
• Nested list widths reveal the response of the complete system to added candidates.

The same position, color, candidate source, and evaluator contract are reused across arms. Pairing removes much of the position-to-position noise that makes two independent game samples hard to compare.

But nested widths still change both list size and distractor composition. That is why we added E023 as a separately versioned, outcome-blind follow-up rather than rewriting E020/E021 after the fact.

Hold the anchor and width fixed

E023 asks a narrower question: if proposer rank 1 remains available and list width stays at five, does the strength of the other four candidates change the model's choice?

The future candidate ranks come from the already registered cached Stockfish top-10 analysis. Rank and score remain hidden from the model; it sees only randomized UCI and SAN moves.

Arm	Hidden proposer ranks	What changes
near_strong	1, 2, 3, 4, 5	Rank 1 competes with four strong alternatives
mixed_tail	1, 2, 8, 9, 10	Ranks 1 and 2 stay fixed; three weaker moves replace ranks 3–5
far_tail	1, 7, 8, 9, 10	Rank 1 stays fixed; every distractor is weaker
best_omitted	2, 3, 4, 5, 6	Availability control; rank 1 is absent

The primary contrast is near_strong − far_tail in evaluator-independent WDL regret. Both lists have five moves and both contain proposer rank 1. The secondary near_strong − mixed_tail contrast is tighter still: ranks 1 and 2 are held constant while only ranks 3–5 are replaced.

This does not make “distractor strength” a perfect scalar treatment. Proposer rank is a source-defined ordering, and particular moves differ tactically and semantically. The eventual analysis therefore retains phase, tacticality, value gaps, selected rank, WDL regret, and catastrophic blunders. Still, the fixed-width, anchor-preserving comparison removes two major alternative explanations that a generic top-k curve cannot.

Random order is not the same as counterbalanced order

Our existing candidate protocol independently randomizes presentation order and shows no scores or ranks. That is necessary, but repeated independent shuffles do not guarantee a useful paired order intervention.

For a two-move list, a second shuffle can reproduce the first with probability one half. In the deterministic E023 audit over 200 abstract positions, the registered independent-shuffle projection produced 96 identical top-2 pairs. Only 52% of candidate-slot assignments moved.

The realized top-5 and top-10 projections had no completely identical pairs, but only 67 and 78 of their 200 pairs were fully deranged. Their maximum rank-by-slot exposure imbalances were 39 and 26 observations. “We randomized twice” therefore does not imply “every candidate moved” or “rank occupied each slot equally.”

E023 uses a two-period cyclic counterbalance instead. Presentation B rotates presentation A by one slot. For every one of the four arms:

• all 200 paired orders differ;
• all 200 are fully deranged;
• 100% of candidate-slot assignments move;
• each hidden rank occupies each of five slots exactly 80 times across the two presentations; and
• maximum rank-by-slot exposure imbalance is zero.

This matters because a position bias can otherwise masquerade as a distractor effect. If one arm happens to place rank 1 earlier more often, an apparent near-versus-far difference may be a serial-position effect. Exact exposure balance integrates the declared slot bias rather than assigning it to one arm.

The construction is deterministic, not outcome-adaptive. The eventual binding will freeze the seed, position-manifest hash, position order, and realized orders before provider calls.

Falsify the instrument before measuring the models

An experiment can be perfectly reproducible and still fail to recover the mechanism it claims to measure. We therefore tested the abstract E023 design on known synthetic selectors before binding any real position.

The audit assigned each proposer rank the monotone utility −log(rank) and computed exact softmax choice probabilities under sharp, diffuse, and primacy-biased selection. These utilities are not assumptions about Luna or DeepSeek. They are controlled inputs used to test whether the instrument responds in the expected direction.

Synthetic selector	Rank-1 probability, near	Rank-1 probability, far	Near minus far
Sharp	0.925624	0.999088	-0.073463
Diffuse	0.437956	0.676147	-0.238191
Diffuse with primacy	0.422457	0.649790	-0.227334

The intended signature appears. Strong nearby alternatives hurt a diffuse selector substantially more than a sharp selector. Adding synthetic primacy does not erase the contrast because rank-slot exposure is exactly balanced.

That result validates the design's behavior under known inputs. It is not a power analysis, not evidence that either evaluated model is diffuse, and not a model result of any kind. The audit made zero provider calls, zero engine calls, read no position manifest, and used zero observed model outcomes.

What would distinguish ideas from vision?

No single cell receives a psychological label. Interpretation comes from a pattern across separately manipulated stages.

Observed pattern	Mechanism made more plausible	What would weaken it
Low self-shortlist rank-1 coverage; high conditional rank-1 selection in every anchor-present E023 arm	Proposal or “ideas” bottleneck	Coverage repairs under a notation or state-format change, suggesting grounding rather than policy recall
High coverage; lower conditional selection with near than far distractors	Comparison-load or “vision” bottleneck	No paired near-versus-far regret difference under the independent evaluator
High paired disagreement when the identical list is deranged	Order-sensitive selection	Stable choices after exact counterbalancing
Deficit disappears when successor states are shown	Internal transition bottleneck	No repair from explicit successors
Further repair when the opponent's best reply is shown	Reply-search bottleneck	Equivalent performance with and without replies
Poor ranking even over fully specified successors	Value or critic bottleneck	Calibrated successor ranking despite poor move-only selection

The last three rows are why “vision-limited” cannot be the final diagnosis. A model may choose badly among move strings because it cannot reliably imagine the resulting board. Another may imagine the board but miss the opponent's reply. A third may calculate both and still compare the resulting positions poorly. E022's successor-state and reply interventions are falsifiers for a pure comparison account.

The design must also allow mixed mechanisms. Low coverage and poor conditional selection can coexist. Assistance may repair generation while exposing a weaker critic. Candidate order may matter only when alternatives are close. The correct output can be a profile rather than a single label.

Outcomes that can prove us wrong

The hypotheses are useful only if awkward results remain publishable.

• If oracle top-2 does not reduce paired regret versus unaided play, the claim that proposal availability repairs the system fails in that population.
• If near_strong does not worsen conditional regret relative to far_tail, E023 does not support the declared distractor-strength mechanism.
• If a width effect disappears after conditioning on value gap, generic “overload” weakens in favor of ambiguity-specific comparison.
• If exact order derangement rarely changes the selected move, presentation instability is not a major explanation.
• If successors or replies remove the deficit, a transition/search mechanism displaces pure comparison failure.
• If effects reverse by model, color, phase, or tacticality, the interaction is the result; there is no license to average it into a universal story.

Illegal and malformed final answers remain model failures and receive the registered maximal-regret treatment. Provider errors are missing observations and may be resampled only to restore a frozen paired cell. There is no outcome-dependent stopping or substitution.

The evidence boundary today

E023 is a preregistered design, not a completed experiment. It is deliberately unbound to real positions. It cannot execute until:

1. the DeepSeek full-game cohort is terminal and content-addressed;
2. the joint Luna/DeepSeek exact and color-mirror leakage audit passes;
3. two cleared-hash Stockfish labeling passes satisfy every stability gate;
4. the final 600-position E020/E021 manifest is frozen;
5. a separate E023 binding names that final hash and freezes its schedule; and
6. provider budget is explicitly authorized.

E023 does not alter E020/E021's locked 600-position sample, 8,800 registered cells, alpha allocation, or primary contrasts. Retrofitting a cleaner idea into an already registered experiment would damage the very identification we are trying to improve.

The eventual primary outcomes are evaluator-independent WDL regret and catastrophic-blunder rate. Proposer-rank selection, order disagreement, legality, latency, tokens, and cost are secondary. Contrasts are within-position and within-model with position-clustered intervals. The near-versus-far primary test uses two-sided alpha 0.05; secondary hypotheses and interactions use Benjamini–Hochberg false-discovery control.

The outcome-blind audit is preserved in controlled_distractor_design_audit.json with report hash 6dda14923d77ebfa1a3824159b505f27b6b1eeca6103b8b1ea74250f9e235f4d. The full gates and estimands are frozen in the E023 preregistration.

What a score can and cannot say

An Elo table is an honest answer to “which complete system won more games under this protocol?” It is not an answer to “what cognitive component failed?” A candidate curve is richer, but it still mixes proposal availability, list width, distractor quality, order, transition, search, and value.

Mechanistic evidence comes from breaking those links one at a time:

• hold the position fixed;
• measure generation separately from conditional selection;
• hold width and the anchor candidate fixed while changing distractor quality;
• show the identical set in guaranteed different, exactly balanced orders;
• reveal successors and replies as explicit falsifiers; and
• evaluate choices with a family independent of the proposer.

Only then can “lack of ideas” and “lack of vision” become testable hypotheses rather than stories attached to the same number.

Scores rank systems. Paired interventions distinguish mechanisms—and the experiment earns its diagnosis only by surviving the outcomes that could prove it wrong.