The Tool Call That Looked Like Causality

A model asked for the board, then made a move. The hard part is proving what the board changed.

Here is a result that practically writes its own misleading headline.

In a frozen corpus of Luna chess transcripts, the model sometimes declined to move immediately and asked to see the current board. Across 13,034 such same-position transitions, its next response contained a legal command accepted by the historical parser 12,379 times: 94.97%.

When Luna asked for the complete legal-move list, the next-response rate was even higher: 621 accepted commands after 623 queries, or 99.68%.

Those numbers are real. They are content-addressed, reproducible, and based on the complete eligible frozen corpus rather than a hand-selected sample.

They still do not show that either tool helped.

That distinction—between a striking chronology and an identified causal effect—is the reason we built two experiments instead of one. A104 reconstructs what happened around historical tool calls. A105 specifies the randomized, byte-matched intervention needed to learn what revealing the same information actually changes.

The bridge between them is a useful case study in evaluating agents. Tool use is not merely an output. It changes the model's next input, occurs at a moment chosen by the model, and often follows some earlier difficulty. If we treat the next successful action as proof that the tool caused recovery, we confuse the agent's policy for an experiment.

First, preserve the chronology exactly

A104 made no provider calls and no chess-engine calls. It read the content-addressed Luna replay corpus already built for the interface studies and verified every frozen source file, response byte range, parser-visible response, and reconstructed chess state before counting anything.

The source contained 27,218 eligible retained response attempts. A104 grouped attempts into chains at the exact same game, ply, and position, then formed only consecutive same-position transitions. That produced:

Frozen A104 quantity	Count
Eligible response attempts	27,218
Consecutive same-position transitions	13,715
Position chains	13,503
Source games	220
Run groups	22

The historical dispatcher had a fixed precedence. It looked for a board query, then a legal-list query, then reflection, before trying to recover a make_move command. A104 replayed that grammar rather than inventing a more convenient interpretation after seeing the outputs.

The immediate transition table was:

Prior response	Same-position transitions	Next accepted legal command	Rate
Board query	13,034	12,379	94.97%
Legal-list query	623	621	99.68%
Non-tool unresolved	58	28	48.28%

The prompt chronology also reconciled exactly. Every board query was followed by the expected board-display shape; every legal-list query was followed by the expected comma-separated legal UCI list. This rules out a mundane bookkeeping story in which we accidentally attached the wrong next prompt.

It does not rule out confounding.

Why 99.68% is not an effect size

The legal-list rate is not the result of randomly assigning some positions to receive a legal list and others not to receive one. Luna chose when to request the list. That choice was part of its behavior.

Several things changed together:

1. The model selected the treatment. A position that triggered a legal-list request may differ systematically from one that triggered a board request or no request.
2. The request came at a particular attempt depth. A tool call could follow an invalid command, uncertainty, or another query. The next response had more interaction history than an immediate move.
3. The returned prompt changed. A board diagram and a complete legal list differ in content, length, notation, and action affordances.
4. Variants were not interchangeable. Candidate-assisted and unaided runs induced different positions and policies. Of 599 board-to-list escalations, 537 occurred in the unaided regular variant.
5. The endpoint was parser resolution. An accepted command is not evidence that the move was good, that the model understood the payload, or that it calculated a better continuation.

Even the 48.28% non-tool comparison is not a control group. It contains only 58 transitions selected by a different historical path. Subtracting 48.28% from 94.97% would produce a large decimal and no defensible causal estimand.

The problem can be written compactly. Let T be the kind of tool query and Y the next accepted command. A104 observes:

P(Y = accepted command | T = board query)
P(Y = accepted command | T = legal-list query)

What we want is closer to a potential-outcome contrast on the same position and same interface:

E[Y(board content revealed) - Y(board content masked)]

The first conditions on a decision made by the historical policy. The second requires us to assign what the model sees.

That is why A104's preregistered decision was not “ship the tool” or “the model used the information.” It was: the census is large and well-reconciled enough to justify a prospective matched ablation.

The control cannot merely be a shorter prompt

A naive follow-up would show one group a board, show another group nothing, and compare their moves. That would randomize something, but not just board information.

The board arm would have a longer prompt, visible spatial structure, a section heading, 64 square slots, and perhaps more time or attention devoted to the position. A legal-list arm would expose not only move identities but also the number of legal moves and their UCI shape. If the arms used different response interfaces, a list could help by making the syntax easier rather than by changing chess judgment.

A105 therefore uses a four-cell factorial with the same panel-shaped payload in every cell:

A105 cell	Board panel	Legal-action panel
no_information	Same-length masks	Same-length masks
board_only	Correct root board	Same-length masks
legal_list_only	Same-length masks	Complete correct legal UCI list
board_plus_legal_list	Correct root board	Complete correct legal UCI list

Hidden board characters become one-byte x tokens. Hidden legal moves become the same number of x characters as the corresponding UCI move. The panel labels, delimiters, 64-square layout, legal-action slot count, move-token lengths, and total UTF-8 byte count remain unchanged within a matched block.

The abstract renderer audit checked 600 position-presentation blocks. It found zero byte-length mismatches, zero masked chess-token leaks, and exactly 179,520 aggregate panel bytes in each of the four conditions.

This is not a claim that byte equality neutralizes every psychological effect. Masked panels are visibly masked. The structural control still reveals the shape of a board, the number of legal actions, and the length of each action token. Those facts are named as retained structural information, not hidden in a footnote.

The point is narrower: the correct board and move identities can be factorially revealed while prompt structure and length remain fixed.

Four cells answer more than three arms

It would be tempting to compare board, list, and neither. Adding the fourth cell—board plus list—matters because the two representations may substitute for or complement one another.

Suppose a board helps only when no legal list is present. That suggests the two panels provide overlapping practical support. Suppose neither helps alone, but both together help substantially. That is a package interaction, not evidence for two independent main effects.

Let R(B,L) be mean WDL regret when correct board content B and correct legal-list content L are either present (1) or masked (0). Lower regret is better. The registered board main effect is:

0.5 × ([R(0,0) - R(1,0)] + [R(0,1) - R(1,1)])

The legal-list main effect is:

0.5 × ([R(0,0) - R(0,1)] + [R(1,0) - R(1,1)])

Each averages the treatment's effect with and without the other panel. The interaction asks whether those simple effects agree.

These expressions are registered before model outcomes. The audit's synthetic surfaces verify that the analysis distinguishes a board-limited signature, a legal-enumeration-limited signature, substitutable representations, complementary representations, and a null. Synthetic recovery validates the instrument under stipulated inputs. It is not evidence that Luna or DeepSeek has any of those mechanisms.

Why the FEN changes the scientific claim

Every A105 arm receives the same full FEN.

That choice is crucial for matching the chess state, but it narrows what the experiment can mean. FEN already specifies piece placement, side to move, castling rights, en-passant target, and move counters. A correct board panel is a different rendering of state already present in the prompt. Legal moves are also derivable from that state and the rules of chess.

So A105 is not a clean experiment in supplying otherwise unavailable Shannon information. It is an experiment in access and representation.

The model may be more effective when a serialized position is expanded into a spatial grid. It may be more effective when legal actions are explicitly enumerated rather than derived. Either effect would matter for agent design. Neither would establish that the model lacked the underlying information in an information-theoretic sense.

This distinction protects against two opposite exaggerations:

• A positive effect would not prove that the model “cannot understand FEN” or that it literally learned new board facts.
• A null effect would not prove that the model internally represents the board perfectly or knows every legal move.

The intervention changes how derivable state is presented. That is the causal object.

Hold the action interface still too

Every A105 cell uses the same full legal-action universe and the same required submit_move tool. Its sole argument is a non-enumerated UCI string.

The schema cannot contain an enum of legal moves. An enum would quietly reveal the legal list in every condition and destroy the list ablation. The historical conversational make_move substring parser is also prohibited as a fallback. If the two provider paths cannot expose the byte-identical tool schema without a candidate menu, the experiment is a no-go.

Illegal submissions, malformed tool calls, refusals, and wrong actions after the registered retry sequence receive maximal regret. They are not silently removed. This means the primary intention-to-treat estimand is a total systems effect. It can include improved execution compliance as well as improved chess choice.

Move quality among legal submissions will be reported secondarily, but that subset is not randomized. We cannot condition away execution failures and then pretend the surviving-move contrast identifies internal use of information.

The 4,800-cell schedule

A105 will bind the 150 development and 150 held-out positions from the eventual final E020/E021 600-position manifest. The 300 training positions are excluded. The design names two model families separately: Luna and DeepSeek V4 Flash.

Each of the 300 positions receives:

• four information conditions;
• two universal panel orders—board first and legal list first; and
• two model families.

300 positions × 4 conditions × 2 panel orders × 2 models = 4,800 cells

That is 2,400 planned requests per model. Every request begins from a fresh context. Condition execution ordinal follows a deterministic balanced Latin rotation: across the abstract schedule, each condition occupies each of the four ordinals exactly 300 times.

Development is for instrumentation and nuisance-variance work. Only untouched held-out positions support primary claims. Primary board and legal-list effects are reported separately for each model, producing four primary tests controlled with Holm's familywise correction. Pairing occurs within position; inference clusters at source game and must show position-cluster and weighting sensitivities.

The planning audit also publishes an MDE envelope rather than declaring the study “powered” by sample size alone. At 150 held-out positions, the normal-approximation MDE ranges from 0.0273 WDL for a paired SD of 0.10 to 0.1091 for a paired SD of 0.40. Treating 75 source-game clusters as a conservative denominator widens that range to 0.0386–0.1542. The actual paired variance, intragame correlation, route behavior, and attrition remain unknown.

An effect must survive attempts to explain it away

A statistically nonzero average is not the only gate. A named representation effect is publishable only if all of the following hold:

1. The held-out model-specific main effect survives the registered multiplicity correction in the beneficial direction.
2. Both simple effects agree in direction—with and without the other panel—so one antagonistic cell is not hiding beneath an average.
3. The conclusion does not materially reverse across panel order, provider route, equal-position versus equal-source-game weighting, position-cluster sensitivity, or maximal-regret attrition bounds.
4. Complete rendered-prompt bytes differ only at registered content offsets.
5. Tool-schema and legal-action-universe hashes match across every arm.
6. Legal-submission and all-request results are shown together without calling the legal-only subset randomized.
7. The effect exists in held-out positions for the named model. A pooled cross-model average cannot substitute for model-specific evidence.

Provider and network failures remain reliability outcomes. A failed request may be retried only in the identical frozen cell under the fixed retry limit. The release must show condition-by-route failure hazards, complete-pair estimates, intention-to-run maximal-regret assignment, best/worst bounds, and the number of positions and source games supporting each contrast.

Differential attrition or a sign reversal under those bounds blocks the causal headline.

What A105 will still not prove

Even a clean positive result would authorize only narrow language:

In the frozen A105 positions and interface, randomized assignment to the named redundant root representation changed total systems WDL regret for the named model snapshot.

It would not prove:

• that A104's historical board or legal-list calls caused Luna to recover;
• that the model acquired otherwise unavailable state;
• that the model consciously read or internally used the panel;
• that the intervention improved search, value estimation, long-horizon play, Elo, or production reliability; or
• that the result generalizes beyond the frozen positions, prompts, providers, routes, models, evaluator, and time window.

Likewise, a null bounded by the registered MDE would not prove that representation never matters. It would be a protocol-specific null within a known precision envelope.

There are no A105 model outcomes in this post.

The current A105 artifact proves that the abstract schedule is balanced, that the panel contents can be byte-matched, that known synthetic factorial signatures are recoverable, and that the experiment has explicit no-go and claim gates. It does not say whether either model benefits from a board or a legal list.

The lesson is larger than chess

Agent traces are full of seductive before-and-after stories.

The agent searched, then answered. It opened a file, then fixed the bug. It retrieved a memory, then completed the task. It asked for the board, then made a move.

Those sequences are operationally valuable. They tell us where the policy chooses help, which tools precede successful actions, and which interventions deserve controlled tests. But an adaptively chosen tool call is not its own control group.

Good agent research needs both views.

The observational trace preserves ecology: what the actual system chose to do. The randomized ablation supplies identification: what changed because a specific payload was revealed under matched conditions. A104 gives us the first. A105 is designed to give us the second.

The dramatic percentages were not the conclusion. They were the invitation to build the experiment that could earn one.

Evidence and reproducibility

This essay is grounded only in the frozen A104 census and the outcome-blind, preregistered A105 design:

• A104 preregistration
• A104 frozen analysis
• A104 reader-facing report
• A104 frozen transition census
• A105 preregistration
• A105 deterministic design audit
• A105 design-audit implementation
• Claims ledger

SHA-256 at drafting:

4cce1ba840369361ac7b4b47ad52c74dd68235ca2abfbfe2b116900011c5dd4d  A104 preregistration
6df7c514431d127957d5f46e36eefb14c2f395dfbf94aada522e8822be70abca  A104 frozen analysis
e73bed27aa5a625e71af1402f95da619dd8529753d3809dca2a4e19ad63c3241  A104 report
deeba48838f8d85ea1fd5f1fdf05d5d8d8ac6bb0ca920591c3b9aabc3e89813b  A104 transition census
f5666d855f847d45f937df31d9b441575b9d4642a6142aaa6ac33796900e297f  A105 preregistration
10c61ed69b02120ef8de35fd490d7573319c90cbad2a6f10a1933985d5fc398a  A105 design audit
09774176d176bdfab9df57be46256705c917efca4dee240ced72fcec2ef8aee1  A105 audit implementation