The Shuffle That Looked Like a Signal

If a language model sees the same chess moves in a different order, does it choose differently?

The naive experiment is easy: show one list, shuffle it, ask twice, and count changes. It is also wrong. A model can change its answer even when the prompt is byte-for-byte identical. Without measuring that baseline instability, ordinary sampling noise masquerades as an ordering effect.

We ran the stronger version. For each of 48 frozen Black-to-move positions, Luna received the same five legal candidates in two visible orders. The second order was a complete cyclic derangement: no move remained in the same slot. Each order was repeated twice in a fresh context. The execution schedule was balanced between ABBA and BAAB blocks and frozen before the first response.

Cross-order decisions disagreed 42.7% of the time. That sounds dramatic until we look at the controls. The two identical A prompts disagreed on 39.6% of positions; the identical B prompts disagreed on 35.4%. After subtracting that same-order instability, the preregistered excess was only 5.2 percentage points. Its bootstrap interval ran from -2.1 to 13.5 points, and the within-position randomization reference was p=0.191.

So the careful result is not “order never matters.” The experiment remains compatible with a modest, potentially meaningful positive effect. It says we could not cleanly separate that effect from Luna's already substantial fresh-context variability with 48 positions and two repetitions per arm.

There was a tempting secondary pattern: the first visible slot was selected 58 times, compared with 38, 25, 29, and 42 for slots two through five. But this was not the primary controlled contrast, and candidate identity still matters. We record it as a follow-up signal, not a headline.

The run also caught a quieter lesson about research engineering. One provider response returned HTTP 200 with a body that was not valid JSON. The frozen runner retried the identical cell and succeeded. Our original locked analyzer, however, treated the presence of any HTTP status as terminal. We discovered that bug from operational metadata before inspecting moves or arm contrasts. Instead of rewriting history, we preserved the original analyzer and froze a versioned amendment. Both analyses remain in the record. Their primary estimate is identical; the corrected version removes one spurious failure.

That is what “real experiment” means here. It is not merely spending tokens or producing a number. It is building the control that can make the obvious number misleading, locking the analysis before reading the result, retaining failures, and saying “not clearly supported” when the interval crosses zero.

E025 does not tell us which move was best. It does not measure Elo, search, or chess vision. It tells us something narrower and useful: in this candidate-menu protocol, most apparent cross-order instability was already present when the order did not change.