An Open Benchmark for Tool-Mediated Chess

A release design for measuring players made of models, prompts, tools, search, and budgets

Release status: v0.1 draft. This document specifies a benchmark; it does not announce a terminal benchmark release. The Luna and DeepSeek ladders are exploratory pilot evidence, not v1 leaderboard submissions. DeepSeek remains live and provisional.

What is the Elo of a language model?

Our experiments suggest that the question is underspecified.

Luna played weakly without candidate help and dramatically better when shown two randomized, unranked Stockfish proposals. Ten proposals barely helped. The requested model was unchanged. The prompt, action interface, candidate oracle, inference budget, color, opponent, and stopping rule defined a different player.

DeepSeek V4 Flash made a second problem visible. Its high-reasoning chess agent could spend hundreds of thousands of tokens per game and hours of occupied worker time while the local processes remained healthy. Its losses included both checkmates and action-protocol forfeits. “Chess strength,” “interface reliability,” and “cost” were different properties of the same submitted system.

A single model-name column cannot represent this evidence.

The LLM Chess Systems Benchmark begins from a different unit:

A submission is the complete decision system: model, provider route, prompts, state representation, legal-action interface, tools, candidate or search policy, inference budget, opponent protocol, data partition, and stopping rule.

The benchmark has two scientific jobs. It must measure whether the system can play complete games. It must also explain which component succeeds or fails on controlled positions. A third track tests architectures that allocate compute differently.

The purpose is not to replace one leaderboard with a larger leaderboard. It is to make every point on the board auditable.

Figure — Benchmark evidence flow

Why the pilots cannot simply become benchmark v1

The Luna and DeepSeek pilots produced valuable discoveries. They also exposed the exact reasons a confirmatory benchmark needs a stricter protocol.

One color and one starting position

The model played Black from the standard initial board. That measures one real full-game system, but it does not estimate a color effect or generalization over openings. Separate conditions can enter correlated opening funnels.

Candidate sets were not guaranteed to be nested

Each candidate width used a separate 0.1-second MultiPV call. The source engine and nominal time were matched, but top-2 was not guaranteed to be an exact prefix of top-10. The observed Luna curve is a system result, not yet a causal effect of list width.

Wall-clock engine work was load-dependent

Several suites shared one host. A 0.1-second search can reach different depths under different machine load. Confirmatory advisor and scoring services require fixed nodes, fixed threads, fixed hash, pinned binaries, and cached outputs.

Interface losses and chess losses were bundled

A wrong-action forfeit is a legitimate complete-system loss. It is not the same mechanism as selecting a legal move with high WDL regret. Both must remain visible.

Adaptive ten-game blocks were exploratory

The ladder moved between opponent anchors based on observed scores. That is efficient for discovery, but small saturated blocks and evolving rules do not justify narrow, preplanned uncertainty claims.

Maximum-ply draws were administrative

Reaching 200 plies triggered a draw under the pilot protocol. A confirmatory release must preserve that protocol outcome while also publishing engine adjudication, tablebase adjudication where available, and censoring sensitivity.

The benchmark does not discard the pilots. It labels them pilot, freezes their evidence, and uses their failures to design v1.

Three tracks, three questions

The benchmark deliberately refuses to force full games, move diagnosis, and architecture efficiency into one composite score.

Track A — Full-game system strength

Track A asks:

Can this exact submitted system play complete chess games reliably under a frozen opponent and opening protocol?

The unit is a paired two-game mini-match from one frozen opening prefix. The submitted system plays each color once against the same opponent configuration. Mini-matches are randomized and interleaved across provider time. Every move after the opening prefix comes from the submitted system.

Required evidence includes:

• raw win, draw, and loss from the submission's perspective;
• natural termination reason;
• provider-error and interface-forfeit indicators;
• administrative censoring and independent adjudication;
• complete PGN and turn-event lineage;
• opening identifier, system color, and opponent anchor; and
• tokens, calls, latency, worker-hours, engine work, and attributable dollars.

The primary rating is a Stockfish-18 UCI-anchor estimate under the benchmark protocol. It is fit jointly over opponent anchors with a W-D-L likelihood, draw propensity, color effect, opening clustering, provider-day sensitivity, and the declared adaptive allocation rule.

It is not FIDE Elo, Lichess Elo, chess.com Elo, or automatically comparable with the upstream Dragon harness.

The leaderboard always shows raw W-D-L and sample size beside the rating. A saturated score such as 10–0 produces a one-sided bound or promotion decision, not an invented finite Elo.

Track B — Paired-position diagnosis

Track B asks:

On the same position, which intervention changes the model's decision, and which operation does that implicate?

The position—not the individual prompt—is the statistical cluster. Every condition uses one canonical, cached engine analysis and differs only through a preregistered intervention.

Core conditions include:

• unaided move generation;
• a model-generated shortlist;
• deterministic nested oracle top-2, top-3, top-5, and top-10;
• the best move omitted;
• random legal and plausible distractors;
• explicit successor states and opponent replies; and
• repeated presentation-order permutations.

The primary endpoint is WDL regret relative to an independent fixed-node scorer. Catastrophic WDL loss, legal-action rate, best-available selection, and paired position-level differences remain adjacent.

Mechanistic endpoints include self-shortlist recall, oracle-candidate adoption, selected canonical rank, order instability, value-gap interaction, game phase, legal-move count, request latency, tokens, retries, and correction turns.

This is where “too many ideas,” “weak proposer,” “search horizon,” and “state grounding” become falsifiable. A full-game Elo difference motivates those claims; it cannot establish them alone.

Track C — Architecture and compute control

Track C asks:

Can a different division of labor improve the strength–reliability–resource frontier under a matched budget?

Eligible systems include:

• deterministic state services and legal-action masks;
• proposer–selector–value decompositions;
• uncertainty-gated deeper reasoning;
• progressive candidate narrowing;
• explicit search controllers; and
• trained or distilled policy/value adapters.

Every architecture submission must name a budget-matched baseline. A natural mode may be reported, but it cannot prove efficiency by itself. Cutting tokens by removing the ability to solve hard positions is not a systems improvement.

Track C publishes strength, local regret, protocol reliability, tokens, latency, worker-hours, engine nodes, and reconciled dollars as a Pareto set. v1 does not assign weights that collapse those dimensions into one number.

The benchmark data has boundaries

Each release has three disjoint, content-addressed partitions.

Partition	Purpose	Visibility
Development	Instrumentation and prompt debugging	Public
Validation	Architecture and compute-policy selection	Outcomes disclosed after submission
Test	Final claims and leaderboard	Positions withheld until evaluation

Position families are grouped before splitting. Normalized FEN, pawn/material structure, source game, and opening lineage prevent near-duplicates from leaking across partitions.

The registered v1 diagnostic target contains 600 positions:

• 200 tactical, 200 ambiguous, and 200 quiet;
• 40% middlegames, 30% openings or early middlegames, and 30% endgames within each stratum;
• equal side-to-move balance; and
• exclusion of all 25,621 unique normalized Luna-pilot keys reconstructed from 27,284 retained position occurrences.

The 600-position design belongs to preregistered E020/E021. Expanding it to 1,000 positions would create a new benchmark version, not a silent update. The locked v1 schedule uses two total presentation orders for candidate-bearing cells on the 200 ambiguous positions. A four-or-more-order study is a separate preregistration or versioned extension, not an unrecorded expansion of those 8,800 cells.

Withholding exact items is necessary but not sufficient. Chess games, puzzles, openings, and engine labels are widespread public data, and dynamic play does not certify that a proprietary model has never seen related structures. Each release therefore declares source cutoffs and exposure dates, groups complete source families into one partition, filters generated positions against every prior public release, and rotates the withheld test partition only through a new benchmark version. Submissions disclose chess-specific training and retrieval corpora when known. These controls reduce direct contamination; they do not turn an unknowable training history into a proof of purity.

Candidate assistance must be deterministic before presentation

For each diagnostic position, the candidate service performs one fixed-node max-k MultiPV analysis. The canonical record stores:

• engine version and binary SHA-256;
• fixed threads, hash, node budget, and material UCI options;
• source FEN;
• canonical ranks, WDL/scores, nodes, depth, and principal variations; and
• a cache-key digest.

Every smaller set is a prefix of that one max-k analysis. Only after canonical analysis does the benchmark randomize the model-facing order. The presentation seed derives from stable experiment, position, condition, repetition, and prompt identifiers.

Scores, ranks, and principal variations remain hidden unless they are the named intervention. The scorer used for outcome analysis is independently identified when an engine advisor is part of the treatment.

This turns “top-2 versus top-10” into a real width comparison instead of two separate wall-clock searches.

A submission names a system, not a brand

Every entry validates against the versioned JSON transport contract in research/schemas/benchmark_submission.schema.json.

Its identity includes:

• benchmark version, protocol version, and submission ID;
• evidence status: pilot, provisional, or confirmatory;
• requested model, returned model when exposed, provider route, and observation window;
• reasoning effort and candidate/search assistance;
• code commit and dirty-diff digest;
• dependency-lock and prompt digests;
• engine binary, settings, and resource limit;
• dataset, schedule, and preregistration hashes;
• stopping rule and exclusions;
• raw unit counts, primary result, and uncertainty;
• resource-accounting coverage; and
• every released artifact's path and SHA-256.

A simplified identity looks like this:

{
  "benchmark_version": "0.1-draft",
  "submission_id": "example-system-2026-08",
  "evidence_status": "provisional",
  "system": {
    "requested_model": "provider/model",
    "returned_model": null,
    "provider": "declared-route",
    "reasoning_effort": "high",
    "assistance": {
      "kind": "candidate_oracle",
      "candidate_count": 2,
      "scores_visible": false,
      "ranks_visible": false
    }
  },
  "protocol": {
    "track": "full_game",
    "stopping_rule": "declared before collection"
  }
}

The real schema also requires content hashes, timestamps, engine provenance, results, uncertainty, resource fields, and artifact hashes. Schema validity is necessary, not sufficient. A well-formed claim can still be wrong.

Four leaderboards, not one winner

The release presents separate views.

1. Full-game protocol rating

Raw W-D-L, joint local rating, interval, colors, openings, assistance, and evidence status.

2. Paired-position move quality

WDL regret, catastrophic-blunder rate, paired-cell count, cluster uncertainty, and condition contrasts.

3. Reliability and interface adherence

Provider failures, malformed or illegal actions, correction turns, interface forfeits, and administrative endings.

4. Strength–resource frontier

Strength or regret against tokens, latency, API calls, worker-hours, engine nodes, and request-attributed dollars.

Mechanistic signatures by position class appear as analysis views rather than a fifth universal ranking. A model can be a strong selector, weak proposer, slow controller, and reliable interface at the same time. A composite score would hide that structure.

No leaderboard point appears without sample size, raw outcomes or paired-cell count, uncertainty, evidence status, assistance label, color/opening coverage, and the completeness of its resource accounting.

Errors are evidence

Provider failure, interface failure, and chess failure receive different treatments.

Provider failures

They remain in the raw event ledger and reliability table. They are excluded from clean chess estimates, and only the registered missing cell may be replaced. The replacement follows the frozen schedule order.

Interface failures

They are losses in the complete-system endpoint. They are also reported separately so move-quality analysis does not pretend a parser failure is a centipawn error. Illegal or malformed responses after the frozen retry policy are never silently deleted.

Administrative censoring

Maximum-ply endings retain the result specified by the game protocol. The benchmark separately reports fixed-node adjudication, tablebase results where available, and a censoring analysis.

Publishing only whichever treatment raises a submission's rank would be post-hoc protocol selection.

Compute is part of the result

The benchmark reports separately:

• prompt, cached, completion, and reasoning tokens when available;
• API calls, retries, first-token latency, completion latency, and total request latency;
• game wall time;
• aggregate evaluation worker-hours;
• provider-reported model-reply time;
• local engine nodes and elapsed time;
• request-attributed dollars when available; and
• account-level spend under a separately named field.

Five concurrent suites for one hour equal five worker-hours and one hour of user wall time. Neither is provider GPU time.

Account-level OpenRouter usage is not silently assigned to a model. The pilot already demonstrated why: Luna and DeepSeek shared the harness credential, and early monitoring briefly queried an unrelated key. That correction is preserved in the append-only ledger. Model dollars become publishable only after request-level activity is joined to model identity and timestamps.

Open means regenerable, not merely downloadable

A credible release needs more than a zip file of PGNs.

The evidence stack is layered:

1. immutable raw game and turn events;
2. content-addressed cohort manifests;
3. regenerated game and ladder indexes;
4. frozen statistical outputs with declared inputs;
5. figures generated from frozen outputs;
6. claims and decision ledgers;
7. a publication inventory containing hashes, not secret-bearing raw logs; and
8. one clean-environment command that rebuilds the release.

The publication bundle never includes private environment file, credentials, authorization headers, or raw secret stores. Paths that escape the repository, enter _logs directly, or contain forbidden credential-bearing components are rejected by the auditor. Text artifacts are scanned for private keys and common API-token patterns.

Open also means the inconvenient record remains visible:

• provider errors;
• interface forfeits;
• excluded cells;
• protocol deviations;
• invalidated cost snapshots;
• superseded artifact hashes; and
• negative mechanistic results.

Corrections create new manifests. They do not rewrite the past.

The release audit fails closed

The repository contains a machine-readable publication specification and an auditor with two modes.

uv run python research/release/build_publication_bundle.py --mode draft
uv run python research/release/build_publication_bundle.py --mode publication

Draft mode permits named publication-only gaps. Publication mode rejects:

• missing required artifacts;
• drift in pinned frozen files;
• missing generators;
• generated files with stale or unverifiable provenance;
• live evidence that has not been frozen; and
• secret-like material.

The manifest is an index of evidence, not a copy of evidence. It records each artifact's path, role, state, hash, generator, input provenance, freshness, and secret-scan result. The canonical hash of the manifest inventory defines the reviewed bundle.

This is an important behavioral choice: a failed audit still writes its manifest. Release blockers become reviewable evidence rather than an error message lost in a terminal.

v0.1 draft status and blockers

The draft audit currently passes. Publication mode does not. After this manifesto and fingerprint analysis were added, the refreshed publication audit indexed more than thirty declared artifacts, found zero secret-like material, and reported explicit publication blockers. The exact machine-generated counts remain in the manifest so this prose does not become stale as the inventory grows.

The release already has:

• a content-addressed Luna freeze;
• frozen Luna rating, trajectory, and candidate-curve artifacts;
• methods, pilot audit, claims ledger, and decision register;
• benchmark and submission schemas;
• deterministic candidate, event, scheduling, and telemetry infrastructure;
• a secret-scanning, provenance-checking publication auditor; and
• drafts for the research and architecture program.

The current publication blockers are intentionally visible:

1. DeepSeek is not terminal. Active suites remain, so its efficiency, trace, and strength artifacts are live or provisional.
2. DeepSeek has no cohort freeze. A content-addressed terminal manifest cannot be created while evidence is still changing.
3. There is no frozen terminal DeepSeek rating. A provisional perfect lower anchor must not be converted into a finite point estimate.
4. Billing is not request-attributed. Account usage cannot be assigned to a specific model or cohort yet.
5. Four derived artifacts remain live. The strength-efficiency frontier, candidate fingerprint, DeepSeek efficiency snapshot, and DeepSeek trace diagnostics must be regenerated from terminal frozen inputs.
6. Live provenance is intentionally perishable. Every generated DeepSeek derivative is hash-bound to its declared ledger inputs. A new tracker append makes a prior derivative stale until it is regenerated; terminal publication additionally requires binding to the frozen cohort.
7. Publication audit has not passed. Every machine-reported missing, live, and stale finding must clear, then the clean-environment release command must succeed. The prose deliberately does not hard-code a count while collection continues.

This post resolves the missing benchmark-manifesto draft, not the evidence blockers. It is labeled v0.1 draft until the auditor says otherwise.

How a submission enters the benchmark

A future submitter follows an explicit path.

1. Select a benchmark version and track.
2. Bind the system identity, prompt, dependencies, provider route, and engine configuration.
3. Validate the submission schema and secret policy locally.
4. Commit the stopping rule and exclusion policy before test outcomes.
5. Run the signed schedule; never choose test cells adaptively outside the declared allocator.
6. Upload or expose the required immutable evidence objects.
7. Reconcile raw units, errors, results, resources, and artifact hashes.
8. Rebuild the analysis and figures in a clean environment.
9. Pass publication audit.
10. Receive a new evidence object on the leaderboard.

A replication never silently replaces an older result under the same model name. Provider route, time window, prompt, code, or protocol changes create a new submission identity.

Governance after release

Semantic changes increment the protocol version. Data corrections generate new manifests and retain superseded hashes. Leaderboard corrections identify the exact affected submission IDs. Exclusion rules cannot change after outcomes are visible without relabeling the analysis exploratory.

Submitters and model vendors may provide corrections, missing provenance, or replications. They cannot delete valid adverse outcomes from the immutable raw layer.

The claims ledger is reviewed before every release. It separates observed system results, mechanistic interpretations, and architecture proposals. The benchmark should make overclaiming mechanically difficult, not merely editorially discouraged.

What would make this benchmark fail?

The benchmark fails scientifically if paired-position improvements do not predict any complete-game behavior and we nevertheless market them as chess strength. It fails if full-game ratings remain so protocol-specific that repeated identical submissions cannot reproduce within declared uncertainty. It fails if provider drift dominates interleaved comparisons.

It fails operationally if complete provenance is so expensive that only the maintainers can submit, if test positions leak through source families, or if the audit checks formatting without reconciling raw evidence.

It fails conceptually if the separate leaderboards become a hidden composite through selective promotion of whichever metric favors a preferred system.

Each failure has a response:

• test paired-to-game predictive validity;
• publish repeated control submissions;
• include provider-day and route sensitivity;
• provide a reference runner and small conformance suite;
• rotate withheld partitions by version;
• audit counts and hashes, not only JSON shape; and
• preserve all registered primary endpoints.

If those repairs do not work, the honest outcome is a narrower benchmark, not a larger claim.

The first questions v1 should answer

The release order is deliberately scientific.

1. Does Luna's non-monotonic candidate curve replicate with nested sets, both colors, and controlled openings?
2. Is the repair caused by move inclusion, comparative selection, state support, or reduced interface failure?
3. Does the mechanism transfer to DeepSeek and other model families?
4. Can uncertainty-gated proposal and selection preserve strength at a lower token and latency budget?
5. Which successful external scaffold can be distilled into a policy that no longer receives that oracle at inference?

A leaderboard is the last view of those questions, not the first.

From benchmark to cumulative science

Chess is useful because every system encounters the same legal world. That does not make every evaluation comparable. The player still includes everything between model request and committed move.

The LLM Chess Systems Benchmark treats that complexity as the object of study. It names the whole system, separates full-game validity from local diagnosis, keeps reliability distinct from chess regret, and prices computation as part of the architecture.

Most importantly, it preserves the route from raw event to public claim.

The benchmark is not open because anyone can add a row. It is open because anyone can inspect what the row means, regenerate it, challenge it, and submit a better-defined system.

That is the release we are building. v0.1 is the contract. v1 begins only when the evidence passes.

---

Reproducibility note

This manifesto is grounded in research/BENCHMARK_SPEC.md, the versioned submission schema, research/release/PUBLICATION_CHECKLIST.md, the machine-readable publication specification, and the latest draft audit manifest. Numerical Luna and DeepSeek claims remain governed by the frozen pilot artifacts and claims ledger. No terminal DeepSeek result is asserted here.

The publication auditor records hashes and findings but does not copy raw logs or credentials. Draft readiness is not publication readiness. The post and its audit status should be revised only by regenerating the declared evidence chain, not by editing a derived result until it looks complete.