The Analysis That Existed Before the Results

A causal experiment is not preregistered if its most consequential decisions are still waiting inside the analysis script

We had already designed the intervention.

A105 asks whether making a chess board or a complete legal-action list easier to access changes a language-model player's local decision quality. Its four conditions are byte-matched. Its request compiler has been tested on castling, en-passant, promotion, check, both colors, and four mobility regimes. Every arm contains the same full FEN, task, action universe, and response function.

That sounds ready.

It was not.

The preregistration still contained phrases such as “source-game-clustered bootstrap or randomization inference,” “registered maximal regret,” and “material reversal.” Each phrase points in the right direction. None uniquely determines a result.

How many bootstrap replicates? Which interval convention? What numeric regret does a malformed action receive? Does a provider failure count as a bad move or remain missing? How are positions weighted when source games contribute different numbers of positions? What is a material reversal? Which tests share a multiplicity family? What happens when a route-specific estimate changes sign by 0.001?

If those choices are made after Luna or DeepSeek outcomes are visible, the experiment has acquired a hidden policy layer. The treatment can be perfectly randomized while the inference remains opportunistic.

So we built A105-R1: a synthetic, outcome-blind analysis preflight. It contains no Luna result, no DeepSeek result, and no chess-engine judgment. Its job is to make the future analysis exist before the future evidence.

Four cells, one sign convention

The conditions cross visible board content, B, with visible legal-list content, L:

Cell	Board	Legal list
00	masked	masked
10	visible	masked
01	masked	visible
11	visible	visible

The outcome is WDL regret on [0, 1], where lower is better. We froze positive effects to mean regret reduction:

board = 0.5 × ([R00 − R10] + [R01 − R11])
list  = 0.5 × ([R00 − R01] + [R10 − R11])

Each position appears in board-first and legal-first panel orders. We first compute the factorial contrasts inside each order, then average the two orders within the position. Positions receive equal weight. Models are never pooled.

That last sentence matters. A single pooled “LLMs benefit from boards” number could hide a benefit for one system and harm for another. A105 registers four primary tests: board and legal-list effects for Luna, and board and legal-list effects for DeepSeek. Holm's step-down procedure adjusts exactly that family of four.

A failed action is not a failed provider

The primary intention-to-treat path distinguishes three states:

• a successful execution receives its measured WDL regret;
• an illegal, malformed, refused, or wrong action receives maximal regret 1.0; and
• an exhausted provider or network failure remains missing.

This prevents an interface failure from disappearing into a favorable complete-case subset. It also prevents infrastructure missingness from being silently called a chess loss.

The primary estimate uses provider-complete position pairs after assigning maximal regret to execution failures. A successful-execution-only estimate is reported as a selection-sensitive diagnostic, never as a replacement primary result.

Provider missingness gets a different treatment. For every registered position, the analysis assigns each missing regret to zero or one in whichever direction minimizes or maximizes the signed contrast. The resulting bounds are intentionally blunt. They answer a defensible question: how far could the factorial effect move under the most favorable or adverse bounded completion of the cells we intended to collect?

Cluster the uncertainty where dependence enters

Positions inherited from one source game are not independent stories. A game contributes a trajectory, opening lineage, and correlated tactical structure. A request-level bootstrap would pretend otherwise.

A105-R1 fixes a nonparametric source-game cluster bootstrap:

• 2,000 replicates;
• source games sampled with replacement;
• every position from a sampled game travels together;
• equal position weighting inside the resampled set;
• percentile endpoints at 2.5% and 97.5%; and
• root seed 20260807.

Metric sub-seeds are the first eight bytes of SHA-256(root|model|effect), interpreted as an unsigned big-endian integer. The report includes known vectors. For example, synthetic/model-a::board_main_effect maps to 10442658907668369282.

These details are boring in exactly the right way. A reader can rerun the analysis byte-for-byte instead of guessing which random stream or quantile definition we meant.

The synthetic interval width is not a power calculation. It describes a toy fixture whose values we wrote. The real paired variance remains unknown until real outcomes exist.

The bootstrap does not test the null. That distinction matters because an ordinary uncentered bootstrap reproduces the observed effect; counting its draws on either side of zero is not the randomization distribution promised by the preregistration.

A105 therefore freezes a separate source-game cluster sign-flip test. Under a zero-effect symmetric sharp null, all positions inherited from one source game receive one shared sign. With at most 16 source-game clusters, the analyzer enumerates every one of the 2^G assignments. Above that boundary it uses 20,000 deterministic Monte Carlo assignments, the seed SHA-256(root|model|effect|sign_flip), and a plus-one correction. The synthetic fixture has eight clusters, so each model-by-effect test traverses all 256 assignments exactly. Holm adjustment consumes these randomization p-values; the bootstrap is used only for confidence intervals.

Blind the condition labels too

The synthetic event rows never contain strings such as board_only. They carry opaque codes such as A105-XQ7 and A105-MV2. The mapping has its own hash, and the production key must be held separately from the scored event table.

Blinding does not make analysis judgment-free, but it removes an unnecessary way for familiar condition names to influence debugging, exclusions, or interpretation before the primary report is sealed.

The event contract is strict. A duplicate intended cell fails. A missing cell fails grid validation rather than shrinking the denominator silently. Unknown statuses, out-of-range regret, regret attached to a provider failure, an absent regret on success, an unregistered panel order, or a single source-game cluster all fail closed.

“Material reversal” now means a number

Sensitivity language is often where a preregistration becomes elastic.

A105 reports estimates by panel order and by realized route. It also reports the two simple effects behind each main effect. A sign change is flagged as a material reversal only when both estimates have absolute magnitude at least 0.02. A +0.001 versus −0.040 pair is not promoted into a dramatic reversal; one side is too close to zero under the frozen rule.

Simple effects receive one of three mechanical labels:

• stable nonnegative;
• stable nonpositive; or
• discordant.

These gates do not decide whether an effect is scientifically important. They make the descriptive promotion rule inspectable before results arrive.

The synthetic fixture is designed to disagree with itself

The preflight creates 256 blinded cells: two synthetic models, 16 positions, four conditions, and two panel orders. Its cases deliberately exercise:

• a positive board main effect;
• a positive legal-list main effect;
• a null effect;
• an adverse effect;
• an interaction;
• illegal or nonsubmitted actions;
• provider-missing cells;
• route drift;
• a duplicated row;
• an incomplete grid;
• a broken one-cluster analysis;
• a material sign reversal; and
• a subthreshold sign change that must not be called material.

All three publication labels appear in the generated report: beneficial_under_synthetic_fixture, adverse_under_synthetic_fixture, and compatible_with_null_under_synthetic_fixture.

The labels say “under synthetic fixture” because their p-values and intervals have no empirical meaning. They are known-answer tests for code paths. A synthetic adverse result proves the software can publish harm rather than hiding it. A synthetic null proves the report has a non-significant state. A synthetic missing cell proves the bound widens instead of disappearing.

What passed—and what did not happen

The frozen preflight reports:

Quantity	Result
Synthetic models	2
Source-game clusters	8
Positions	16
Blinded cells	256
Bootstrap replicates per model/effect	2,000
Exact sign-flip assignments per model/effect	256
Registered future primary tests	4
Synthetic adversary checks	11
Provider calls	0
Network calls	0
Engine calls	0
Observed model outcomes	0

The status is pass_synthetic_analysis_contract_only.

That status does not mean the board helps, the legal list helps, the experiment is powered, the provider will preserve assignment, or the final position set is valid. It means the analysis machinery behaves deterministically on stipulated inputs and rejects a defined set of malformed ones.

A105 still waits on terminal replication evidence, joint leakage qualification, the final position freeze, a distinct evaluator identity, realized request auditing, and paid-call authorization. The local request compiler and the synthetic analyzer are necessary gates, not substitutes for the campaign.

The deeper lesson

Preregistration is often treated as a document. For computational experiments, the stronger object is a program that already knows how to be disappointed.

It knows how to report a null. It knows how to report harm. It knows what an illegal action costs. It knows which missingness stays missing. It knows which tests share a family. It knows the seed before it sees a score. It knows that a route-specific reversal smaller than the registered threshold does not become a headline.

This does not eliminate researcher judgment. It moves the most consequential judgments to a time when the answers cannot influence them.

The analysis now exists. The results do not.

That is the point.

Reproduce the preflight

From the repository root:

uv run python research/analysis/a105_factorial_analysis_preflight.py
uv run pytest -q tests/test_a105_factorial_analysis_preflight.py

The frozen contract is A105_FACTORIAL_ANALYSIS_PREFLIGHT_PROTOCOL.md. The generated report is analysis.json, and the blinded synthetic event ledger is synthetic_cells.jsonl.

Every artifact preserves the same boundary: stipulated synthetic values only, zero live model evidence, and zero causal claims.