Chess, Scaffolded Reasoning Laboratory
Menu

Experiment E028 / terminal paired aggregate (final combined result from matched positions)

The prompt did not whisper.
It pulled.

Across the same 48 chess positions, the treatment arm (fixed set shown) reached 95.83% set membership (set adherence: whether the selected action was in the displayed set) versus 2.08% in the board-only control (fixed set not shown): +93.75 pp.

48 paired positions192/192 terminal model actions (final submitted actions)0 provider-error (failed provider response) · 0 unattempted · 0 incomplete positions

Replay the registered contrast

Treatment · five moves visible

Treatment shows one fixed five-move set; control omits it. Both arms retain the same unrestricted move submission. The arm comparison below remains visible as the static reading if this control is unavailable.

95.83%Treatment · 92 of 96 terminal actions inside the fixed five-move set
Control · board only
2 of 96 in set2.08%
Treatment · fixed five visible
92 of 96 in set95.83%
Paired effect (treatment-minus-control difference across matched positions)+93.75 pp

Treatment minus control; position-weighted fixed-set membership.

Position bootstrap 95% (uncertainty range from resampling positions)+88.54 pp to +97.92 pp

Position-level percentile interval.

Within-position randomization (arm labels reassigned within each matched position)p=0.0000099999

Registered two-sided test (specified before outcomes; its p-value measures how unusual the observed difference would be with no arm difference).

What this establishes

Guidance is causal.
Quality is unmeasured.

The treatment arm landed inside the displayed set 95.83% of the time (92 of 96 terminal actions); the board-only control did so 2.08% (2 of 96). Only the fixed set's visibility changed between arms; the submitted action remained unrestricted.

The paired estimate is +93.75 pp, with a position-bootstrap 95% interval of +88.54 pp to +97.92 pp and a registered two-sided randomization p-value of 0.0000099999.

Evidence boundary. Development-only causal effect of showing one fixed five-move suggestion set on Luna set adherence; no move-quality, Elo, vision, search, or candidate-generation claim. Open C025 evidence boundary

We can sayShowing the fixed set causally changed set membership under this prompt and route.
We cannot sayIt improved move quality, chess strength, search, vision, candidate generation, or Elo.

Limits carried into the page

  • The outcome is membership in one fixed five-move suggestion set, not chess strength or move quality.
  • The experiment used one dated Luna model snapshot and development-exposed Black positions.
  • The board-only arm's low legal-action rate is itself part of the measured prompt-format behavior; it is not an Elo estimate.
  • No E027 action or outcome was imported into this fresh response cohort.
Read the full laboratory note →