Elo Is a Property of a Protocol

Why a language model does not have one context-free chess rating

Ask for a chess player's Elo and the question sounds ordinary. Ask for a language model's chess Elo and almost every noun becomes ambiguous.

Which model snapshot? Which prompt? Does it see the board as FEN or Unicode? Can it request legal moves? Does it receive engine candidates? How much reasoning is allowed? Which color does it play? Which openings? What does the opponent's rating implementation mean? What happens after 200 plies? Does an invalid tool call count as a chess loss?

Change any of those and the player changes.

That is not an excuse to avoid ratings. It is a reason to name the object being rated correctly. In agent experiments, Elo belongs to a protocol:

rating = f(model, prompt, tools, inference budget, opponent,
           color, openings, stopping rule, provider snapshot, hardware)

The model is one coordinate in that function.

What our number means

Our Luna pilot has a narrow operational definition:

• Luna played every model move as Black;
• Stockfish 18 played White using UCI_LimitStrength anchors;
• Stockfish received 0.1 seconds per move;
• assisted variants saw randomized, unranked UCI and SAN candidates;
• the candidate source also used a 0.1-second Stockfish MultiPV search;
• model reasoning effort was high;
• every game began from the initial position; and
• 200 plies produced an administrative draw.

“Luna top-2 Elo” is shorthand for the performance of that entire system under those rules. It is not FIDE Elo, Lichess Elo, chess.com Elo, the upstream Dragon leaderboard, or Luna's intrinsic chess intelligence.

The tempting one-line calculation

At one Stockfish anchor, define score as:

S = (wins + 0.5 × draws) / games

The familiar logistic conversion is:

rating = opponent + color correction + 400 × log10(S / (1 - S))

This transform is useful for quick ladder decisions. It produced the initial headline estimates: regular near 1164 and top-2 near 2803 at their most informative anchors.

But the shortcut throws away information.

First, a draw is treated as exactly half a win. A 0–5–5 record and a 2–1–7 record can have similar point scores while implying different win/draw/loss behavior. Second, choosing one “best” anchor ignores the other games the system played. Third, a perfect score produces infinite Elo. Fourth, a fixed color correction looks more certain than the Black-only experiment can justify.

So we replaced the publication estimate with a joint likelihood.

A joint W-D-L model

The new analysis fits every completed Luna block simultaneously using a Davidson win-draw-loss model. Each candidate condition receives one latent local rating. All conditions share an estimated draw propensity. Stockfish anchors remain fixed at their configured UCI ratings.

For model strength ratio x and draw parameter ν:

P(win)  = x / (x + 1 + ν√x)
P(draw) = ν√x / (x + 1 + ν√x)
P(loss) = 1 / (x + 1 + ν√x)

The fit preserves wins, draws, and losses as different observations. It also uses all anchors rather than selecting the block that yields the nicest point.

The resulting exploratory estimates are:

Condition	Games	Combined W-D-L	Tested anchors	Joint local Elo	Profile 95% interval
Regular	10	0–5–5	1320	1067	705–1351
Top 10	10	2–2–6	1320	1127	805–1407
Top 5	60	28–14–18	1320–1920	1830	1713–1950
Top 3	80	43–22–15	1320–2520	2431	2317–2548
Top 2	60	41–11–8	1320–2820	2832	2680–2992

The model estimates a draw parameter of 1.02. Intervals are profile-likelihood intervals: for each candidate condition, its rating is fixed at a sequence of values while every other rating and the draw parameter are reoptimized.

These intervals are not magic shields against the pilot design. They represent sampling uncertainty under the fitted likelihood, not uncertainty about provider drift, adaptive anchor selection, one opening, or the candidate harness itself.

Why regular moved from 1164 to 1067

The earlier 1164 estimate converted the regular block's 25% point score directly. The joint model sees something more specific: zero wins, five draws, and five losses. Given the draw tendency learned across the cohort, a system that draws half and loses half but never wins has a lower maximum-likelihood strength than the half-point shortcut suggests.

Neither value is fraudulently “wrong.” They answer different statistical questions. The mistake would be publishing one without its estimator.

The same distinction matters when comparing leaderboards. Two sites can observe identical games and report different ratings because they use different draw models, priors, pools, color adjustments, or anchoring players.

Perfect scores are bounds

At Stockfish 1320, Luna top-2 and top-3 both began 10–0–0. Top-2 later repeated the feat at 2420. A naive Elo transform tries to divide by zero and returns infinity.

Infinity is not evidence that the player has infinite strength. It means that the tested anchor supplied no upper bracket.

For a 10–0–0 block, the exact 95% score interval is approximately 69.2% to 100%. Under the pilot's 35-point Black correction, 10–0–0 against 2420 provides a local lower rating bound near 2595 and no finite upper bound from that block. The ladder must move to a stronger anchor until losses or draws identify the other side.

This is why provisional perfect scores should be displayed as arrows or bounds, never as invented finite dots.

The color correction cannot be learned here

The pilot adds 35 points because the model always played Black. That reproduces the historical reporting convention, but the experiment contains no White Luna games. Consequently, color strength and model strength are mathematically confounded.

Refitting with three assumed Black penalties demonstrates the problem:

Assumed Black penalty	Regular	Top 10	Top 5	Top 3	Top 2
0	1032	1092	1795	2396	2797
35	1067	1127	1830	2431	2832
70	1102	1162	1865	2466	2867

Every rating shifts by exactly the assumed correction. The games cannot choose among those rows.

This sensitivity does not erase the candidate curve: the relative differences remain. It does prevent us from pretending that the absolute scale is known to 35-point precision. Confirmatory games need color-reversed mini-matches.

Opponent Elo is also an implementation

Stockfish's UCI_LimitStrength=1800 is not a human with an 1800 federation rating. It is a particular engine binary running a strength-limiting procedure under particular time and hardware conditions.

The pilot adds another complication: the opponent receives wall-clock time, so five concurrent suites can change how many nodes Stockfish searches. The configured Elo stays fixed while the realized computation may move.

Confirmatory experiments therefore use:

• a pinned engine binary and checksum;
• fixed nodes, threads, and hash instead of wall-clock limits;
• a frozen opening suite;
• both colors;
• randomized interleaving across conditions;
• engine adjudication plus cutoff sensitivity; and
• an explicit mapping study before relating local ratings to another pool.

An anchor is only as stable as its implementation.

Should we skip the easy anchors?

Only prospectively—and only with an independently chosen starting point.

Ten-game blocks are noisy. Our live ladder promotes by 400, 300, or 200 points after scores of at least .90, .80, or .65; middling blocks receive one repeat, and low scores stop. That saves games compared with exhaustively testing every anchor, but the next opponent is selected by the same outcomes later used for rating. Throwing away the early blocks after the ladder climbs would compound that selection: it keeps the block we happened to stop near and deletes valid evidence about the same protocol player.

We therefore built a deterministic, provider-free audit. It generates W-D-L blocks at known ratings and draw propensities, runs the exact live transitions, then compares three analyses: the current likelihood over every anchor, the same likelihood after deleting all anchors below the final one, and the familiar logistic conversion of only the last block. It also checks the current profile interval, saturated paths, estimator bounds, games spent, and anchor visits.

The point is not to bless the ladder. The simulator generates from the same Davidson family it fits, treats the 35-point Black correction and Stockfish anchors as known, and its ordinary profile intervals do not correct for optional stopping. This is a favorable design check. Under that deliberately matched model, the last block is still a fragile summary: it was selected by a threshold and contains only ten games. Earlier blocks can help the joint likelihood even when some were nearly saturated. That is not a universal proof that low anchors are efficient under model misspecification.

For a new model family, an independently calibrated pilot could justify starting closer to the expected rating. That would save easy games without deleting data after seeing it. It must be recorded as a new protocol, because starting at 1320 and starting at 2200 are different measurement procedures.

The prospective simulation makes the tradeoff concrete. Averaged over its deliberately uniform known-truth grid, starts of 1320, 1720, and 2120 used 39.88, 29.47, and 22.37 games, respectively—but their mean rating RMSE rose from 113.19 to 277.00 and 477.62. The high starts were efficient when the simulated player really was strong and disastrous when it was weak. Against a true 1100-rated player, a 2120 start stopped after one block but produced roughly 578--607 Elo RMSE and 67%--87% saturated blocks. Against a true 3200-rated player, it saved roughly 20 games with broadly similar RMSE to starting at 1320.

Even an impossible oracle start near the true Black-adjusted rating was not an RMSE oracle: the policy often collected only two near-even blocks, too little to estimate rating and draw propensity precisely. There is no context-free best start. The defensible rule is to choose prospectively from independent evidence, price the risk of overshooting a weak model, version the protocol, and retain every resulting block.

A rating report should be a tuple

Instead of writing “Model X is 2400 Elo,” a reproducible report should expose:

(model snapshot,
 harness version,
 assistance condition,
 opponent binary and limiter,
 openings and colors,
 inference budget,
 stopping/adjudication,
 estimator,
 date)

The number can remain in the headline. The tuple must remain close enough that another researcher can reproduce what the number refers to.

At minimum, every rating figure should include raw W-D-L, game count, tested anchors, uncertainty, and whether any score was saturated. A reader should not need to reverse-engineer the experimental unit from a footnote.

What would change our mind?

The protocol-specific view would weaken if the same model produced stable ratings across prompts, board representations, legal-move interfaces, reasoning budgets, colors, opening suites, opponent engines, and stopping rules. That is an empirical possibility, but it should be demonstrated rather than assumed.

It would also weaken if local Stockfish ratings mapped consistently onto human or public-engine pools across many reference agents. We have not established that map.

For now, the evidence points in the opposite direction. Two changes to the candidate interface moved estimated strength by well over a thousand points. The harness is plainly inside the object being rated.

The useful interpretation

“Elo belongs to the protocol” does not make Elo meaningless. It makes Elo an experimental measurement rather than a personality trait.

Within a stable protocol, the candidate curve is striking. Across protocols, the numbers require a bridge study. Across time, provider snapshots require dates. Across colors, the correction requires actual paired games.

The honest claim is precise and still surprising:

Under the frozen exploratory Black-only Stockfish protocol, the joint W-D-L model estimates Luna top-2 near 2832 local Elo, compared with 1067 unaided, with substantial uncertainty and assistance explicitly included in the player.

Precision does not weaken the result. It tells us what result we actually have.

---

Reproducibility note

research/analysis/rating_model.py reads all completed Luna blocks and the content-addressed Luna freeze. It produces research/analysis/luna_joint_rating.json, including raw W-D-L, anchors, profile intervals, shared draw propensity, color sensitivity, and exact bounds for saturated decisive blocks. The analysis is exploratory because the anchors were selected adaptively and all games used one starting position with Luna as Black.