Ideas or Vision? Why the Score Alone Cannot Tell

Suppose two agents find Stockfish's preferred move 31.5% of the time.

The first agent rarely proposes that move. It includes the move in only 35% of its shortlists—but when the move is present, it selects it 90% of the time. The second agent usually proposes the move. It includes it 90% of the time—but selects it only 35% of the time.

0.35 × 0.90 = 0.315
0.90 × 0.35 = 0.315

Their headline accuracy is identical. Their engineering diagnoses are almost opposites.

The first system is ideas-limited in an operational sense: a strong action often fails to enter its consideration set. The second is vision-limited in an equally operational sense: a strong action is available, yet the system often fails to distinguish it from alternatives. One needs better proposal recall. The other may need a sharper critic, less comparison load, more stable presentation, or explicit state transitions and reply search.

Those labels are not claims about human cognition. “Vision” here does not mean that a language model sees a chessboard as a person does. It is shorthand for observable success at choosing among already available actions. Even that shorthand is provisional until further interventions separate comparison, state transition, search, and value estimation.

This is the central measurement problem behind our next paired-position experiments. Full-game Elo can tell us which complete system won more often. Candidate assistance can tell us that changing the interface helped. Neither, by itself, identifies why.

Figure — E023 observational alias and order-control design

Figure: outcome-blind synthetic design evidence. The diagram contains no Luna or DeepSeek mechanism outcomes; it shows why the next intervention is identifiable under its declared inputs.

One outcome, two hidden stages

A useful first approximation treats move choice as a two-stage process:

1. Generation: does the candidate set contain a sufficiently strong move?
2. Selection: conditional on that move being available, does the model choose it?

Let C be best-move coverage and S be conditional best-move selection. Then the observed best-move rate is

P(best move selected) = C × S

Observing only the product does not identify its factors. This is observational aliasing: different latent mechanisms produce the same measured outcome.

The problem is worse in full games. A loss may begin with a missed candidate, a mistaken comparison, an inaccurate imagined successor, a missed opponent reply, a stale board representation, or a later error caused by an earlier position. Adaptive opponent ratings, different openings, and unpaired game trajectories add still more variation. Elo intentionally compresses all of that into one systems-level number.

Compression is valuable for a leaderboard. It is inadequate for mechanism.

Even a top-2 versus top-10 comparison remains aliased. The larger list changes at least three things simultaneously:

• the number of comparisons;
• the quality and ambiguity of the additional candidates; and
• the realized positions and downstream game trajectories.

If top-10 performs worse, we have learned that the complete top-10 intervention was worse. We have not yet learned that the model was overwhelmed merely by counting to ten.

The paired intervention ladder

The prospective E020/E021 design is intended to attack the first layer of aliasing by showing the same position set under different information conditions once its final manifest is frozen. Its planned arms include unaided play, self-generated shortlists, nested oracle lists, and a best-move-omitted control.

That creates diagnostically different observations:

• Self-shortlist coverage asks whether the model can generate a strong candidate.
• Oracle-list selection asks whether it can choose that candidate when an external proposer guarantees availability.
• Best-move omission measures behavior when the anchor action is genuinely unavailable rather than merely ignored.
• Nested list widths reveal the response of the complete system to added candidates.

The registered design will reuse the same position, color, candidate source, and evaluator contract across arms once the final manifest and evaluator inputs are frozen. Pairing is intended to remove much of the position-to-position noise that makes two independent game samples hard to compare.

But nested widths still change both list size and distractor composition. That is why we added E023 as a separately versioned, outcome-blind follow-up rather than rewriting E020/E021 after the fact.

Hold the anchor and width fixed

E023 asks a narrower question: if proposer rank 1 remains available and list width stays at five, does the strength of the other four candidates change the model's choice?

The future candidate ranks come from the already registered cached Stockfish top-10 analysis. Rank and score remain hidden from the model; it sees only randomized UCI and SAN moves.

The primary contrast is near_strong − far_tail in evaluator-independent WDL regret. Both lists have five moves and both contain proposer rank 1. The secondary near_strong − mixed_tail contrast is tighter still: ranks 1 and 2 are held constant while only ranks 3–5 are replaced.

This does not make “distractor strength” a perfect scalar treatment. Proposer rank is a source-defined ordering, and particular moves differ tactically and semantically. The eventual analysis therefore retains phase, tacticality, value gaps, selected rank, WDL regret, and catastrophic blunders. Still, the fixed-width, anchor-preserving comparison removes two major alternative explanations that a generic top-k curve cannot.

Random order is not the same as counterbalanced order

Our existing candidate protocol independently randomizes presentation order and shows no scores or ranks. That is necessary, but repeated independent shuffles do not guarantee a useful paired order intervention.

For a two-move list, a second shuffle can reproduce the first with probability one half. In the deterministic E023 audit over 200 abstract positions, the registered independent-shuffle projection produced 96 identical top-2 pairs. Only 52% of candidate-slot assignments moved.

The realized top-5 and top-10 projections had no completely identical pairs, but only 67 and 78 of their 200 pairs were fully deranged. Their maximum rank-by-slot exposure imbalances were 39 and 26 observations. “We randomized twice” therefore does not imply “every candidate moved” or “rank occupied each slot equally.”

E023 uses a two-period cyclic counterbalance instead. Presentation B rotates presentation A by one slot. For every one of the four arms:

• all 200 paired orders differ;
• all 200 are fully deranged;
• 100% of candidate-slot assignments move;
• each hidden rank occupies each of five slots exactly 80 times across the two presentations; and
• maximum rank-by-slot exposure imbalance is zero.

This matters because a position bias can otherwise masquerade as a distractor effect. If one arm happens to place rank 1 earlier more often, an apparent near-versus-far difference may be a serial-position effect. Exact exposure balance integrates the declared slot bias rather than assigning it to one arm.

The construction is deterministic, not outcome-adaptive. The eventual binding will freeze the seed, position-manifest hash, position order, and realized orders before provider calls.

Falsify the instrument before measuring the models

An experiment can be perfectly reproducible and still fail to recover the mechanism it claims to measure. We therefore tested the abstract E023 design on known synthetic selectors before binding any real position.

The audit assigned each proposer rank the monotone utility −log(rank) and computed exact softmax choice probabilities under sharp, diffuse, and primacy-biased selection. These utilities are not assumptions about Luna or DeepSeek. They are controlled inputs used to test whether the instrument responds in the expected direction.

The intended signature appears. Strong nearby alternatives hurt a diffuse selector substantially more than a sharp selector. Adding synthetic primacy does not erase the contrast because rank-slot exposure is exactly balanced.

That result validates the design's behavior under known inputs. It is not a power analysis, not evidence that either evaluated model is diffuse, and not a model result of any kind. The audit made zero provider calls, zero engine calls, read no position manifest, and used zero observed model outcomes.

What would distinguish ideas from vision?

No single cell receives a psychological label. Interpretation comes from a pattern across separately manipulated stages.

The last three rows are why “vision-limited” cannot be the final diagnosis. A model may choose badly among move strings because it cannot reliably imagine the resulting board. Another may imagine the board but miss the opponent's reply. A third may calculate both and still compare the resulting positions poorly. E022's successor-state and reply interventions are falsifiers for a pure comparison account.

The design must also allow mixed mechanisms. Low coverage and poor conditional selection can coexist. Assistance may repair generation while exposing a weaker critic. Candidate order may matter only when alternatives are close. The correct output can be a profile rather than a single label.

Outcomes that can prove us wrong

The hypotheses are useful only if awkward results remain publishable.

• If oracle top-2 does not reduce paired regret versus unaided play, the claim that proposal availability repairs the system fails in that population.
• If near_strong does not worsen conditional regret relative to far_tail, E023 does not support the declared distractor-strength mechanism.
• If a width effect disappears after conditioning on value gap, generic “overload” weakens in favor of ambiguity-specific comparison.
• If exact order derangement rarely changes the selected move, presentation instability is not a major explanation.
• If successors or replies remove the deficit, a transition/search mechanism displaces pure comparison failure.
• If effects reverse by model, color, phase, or tacticality, the interaction is the result; there is no license to average it into a universal story.

Illegal and malformed final answers remain model failures and receive the registered maximal-regret treatment. Provider errors are missing observations and may be resampled only to restore a frozen paired cell. There is no outcome-dependent stopping or substitution.

The evidence boundary at the 2026-08-05 design boundary

At the 2026-08-05 design boundary, E023 is a preregistered design, not a completed experiment. It is deliberately unbound to real positions. It cannot execute until:

1. the DeepSeek full-game cohort is terminal and content-addressed;
2. the joint Luna/DeepSeek exact and color-mirror leakage audit passes;
3. two cleared-hash Stockfish labeling passes satisfy every stability gate;
4. the final 600-position E020/E021 manifest is frozen;
5. a separate E023 binding names that final hash and freezes its schedule; and
6. provider budget is explicitly authorized.

E023 does not alter E020/E021's registered 600-position sample, 8,800 registered cells, alpha allocation, or primary contrasts. Retrofitting a cleaner idea into an already registered experiment would damage the very identification we are trying to improve.

The eventual primary outcomes are evaluator-independent WDL regret and catastrophic-blunder rate. Proposer-rank selection, order disagreement, legality, latency, tokens, and cost are secondary. Contrasts are within-position and within-model with position-clustered intervals. The near-versus-far primary test uses two-sided alpha 0.05; secondary hypotheses and interactions use Benjamini–Hochberg false-discovery control.

The outcome-blind audit is preserved in [local path withheld] (In-page hash only) with report hash [private hash withheld]. The full gates and estimands are frozen in the E023 preregistration (In-page hash only).

What a score can and cannot say

An Elo table answers which complete system won more games under this protocol; it cannot identify the failed component. Candidate curves still mix proposal availability, list width, distractor quality, order, transition, search, and value. The prospective intervention ladder makes those links testable by holding positions fixed, separating generation from conditional selection, varying distractor quality at fixed width and anchor, counterbalancing identical orders, revealing successors and replies as falsifiers, and evaluating choices with a family independent of the proposer. Until those interventions produce admissible outcomes, “lack of ideas” and “lack of vision” remain hypotheses, not diagnoses.

Scores rank systems. Paired interventions distinguish mechanisms—and the experiment earns its diagnosis only by surviving the outcomes that could prove it wrong.