The Result That Refused to Exist

A research pipeline should make it impossible to publish a conclusion while its evidence is still moving

The most important output of our DeepSeek chess evaluation does not exist.

There are game files. There are live status tables. There are provisional scores, token totals, runtime estimates, and plots. But there is no terminal DeepSeek cohort, no terminal DeepSeek rating, and no publication-grade cross-model frontier. At the 2026-08-04 drafting boundary, E001 still had live evaluation suites and research/data/deepseek_pilot_freeze.json did not exist.

That absence is deliberate.

Most research pipelines make premature claims socially forbidden: the analyst is expected to remember that a table is provisional, that one job is still running, or that a CSV can change after a figure is rendered. We wanted a stronger contract. While evidence is live, the terminal result should be computationally unavailable. A command asking for it should fail. A derivative should refuse an unverified source. A publication bundle should remain blocked.

This is fail-closed terminalization: uncertainty about liveness, membership, protocol, or provenance resolves to “no result,” not to the most convenient interpretation of whatever files happen to be present.

The implementation is the DeepSeek freeze gate, its terminal runbook, and a focused adversarial test suite. The design is specific to this experiment, but the principle generalizes to any evaluation whose artifacts arrive asynchronously.

A live view is not a result

It helps to separate four objects that are often collapsed into one:

Object	Purpose	May change?	Supports terminal claims?
Live status	Operate and monitor the experiment	Yes	No
Canonical ledger	Reconcile completed and active processes	Yes	No
Content-addressed cohort freeze	Declare exact terminal evidence membership	No	Yes
Freeze-bound derivative	Estimate a registered quantity from that exact membership	No, after release	Yes

A provisional dashboard can be accurate at its timestamp and still be the wrong source for a terminal conclusion. Suppose nine games have completed and the tenth changes a score from 0.61 to 0.65. In our adaptive ladder those values imply different actions. One calls for a repeat at the same anchor; the other calls for advancement by 200 Elo. The last file does not merely refine a decimal. It changes which evidence should exist next.

The same problem appears when a completed ten-attempt process contains a provider error. The error is a real intention-to-run outcome, but it is not a clean chess game. The ladder requires only enough supplemental games to reach ten clean observations at that anchor. Freezing the process before the supplement changes the denominator; silently discarding the error changes the reliability record.

“The folder looks finished” is therefore not a terminality rule.

First prove that nothing is running

The freeze checks liveness before reading evidence and checks it again before publication. Each check uses two independent views.

The first is GNU Screen. Every managed evaluation has a persistent session whose name begins with deepseek-v4-flash-. If any such session exists, the freeze stops before opening the ledger.

Screen alone is insufficient. A harness can be launched from another terminal, a session wrapper can race with its child process, or an operator can abandon the naming convention. The second guard inspects the process table for the DeepSeek candidate harness. It returns only a count. Command lines are examined in memory but never copied into an exception or manifest, because a shell command may contain sourced configuration syntax.

The dual guard gives us two failure detectors:

managed session exists  -> refuse
matching harness exists -> refuse
either check unavailable -> refuse

It is not a proof that no process anywhere in the universe can alter a file. It is a concrete defense against the two process surfaces used by this laboratory. File and ledger race checks provide the next layer.

Membership must be exact

A cohort is a set, not a glob.

For each ledger row, the freeze derives the only valid directory from opponent Elo, variant, cohort suffix, and run name. It rejects a row whose declared path does not equal that canonical path. It then compares the ledger's run-directory set with every immediate DeepSeek run directory actually present under the log root.

Equality is required in both directions:

• a ledger path without a directory is missing evidence;
• a directory without a ledger row is an unindexed cohort member;
• a repeated path is duplicate evidence, even if set comparison would hide it;
• a nested directory, symlink, unexpected JSON name, or unrecognized file type is ambiguous membership.

All of these conditions block the freeze.

This is stricter than hashing every file returned by a recursive glob. A glob can faithfully hash the wrong population. Content addressing becomes scientifically useful only after membership itself has been justified.

The accepted inventory contains run metadata, aggregate output, individual game JSON, and known run logs. Every member receives a relative path, byte count, kind, and SHA-256 digest. A canonical serialization of that ordered inventory produces the cohort content-set hash. Added, removed, renamed, or changed bytes produce a different identity.

Reconstruct the result from the lowest-level evidence

The tracker and aggregate are indexes. They are not independent witnesses.

For every individual game JSON, the freeze checks the Black model identity, terminal reason, winner vocabulary, and model-scoped token accounting. It then reconstructs:

• clean wins, draws, and losses;
• terminal provider-error count;
• prompt tokens;
• completion tokens; and
• total tokens.

Prompt plus completion tokens must equal total tokens. Clean W-D-L must equal the ledger's clean-game count. Clean games plus provider errors must equal the process target. The aggregate's Black wins, White wins, draws, game count, and embedded run metadata must agree with the same evidence.

The direction of trust matters:

raw games -> reconstructed process totals -> ledger and aggregate agreement

not:

ledger says 6-1-3 -> assume the ten files say 6-1-3

This protects against stale aggregation, a duplicated ledger row, a malformed game, or a token total copied from the wrong process. It also keeps provider errors visible without allowing them into the clean chess denominator.

Secret safety is part of provenance

Freezing raw evidence can accidentally freeze credentials.

The run metadata is supposed to contain the literal value REDACTED for its API key. The gate recursively checks credential-shaped JSON fields and refuses non-redacted values. It also scans every member for known OpenRouter and OpenAI key prefixes and environment-key assignments.

On a match, the error reports only that credential-like material was detected and names the file. It does not print the match, line, or surrounding text. This distinction is covered by a regression test containing a synthetic secret whose unique body must never appear in the exception.

A hash is not sanitization. Publishing the SHA-256 of a secret-bearing file may avoid printing the secret in one manifest, but it still blesses the unsafe raw member as part of the release. The safe behavior is to refuse the cohort until the source is redacted through an explicit, auditable procedure.

The scanner is intentionally described as a known-marker defense, not a proof that arbitrary sensitive strings cannot exist. Publication still requires independent secret scanning of the complete bundle.

Terminality is a history, not a final score

Even perfect file reconciliation does not establish that an adaptive ladder is finished. The freeze reconstructs ten-clean-game blocks in chronological order and replays the registered policy.

Every variant must start at Stockfish 1320. At each anchor:

• score at least 0.90 authorizes +400;
• score from 0.80 through below 0.90 authorizes +300;
• score from 0.65 through below 0.80 authorizes +200;
• score from 0.35 through below 0.65 authorizes one repeat at the same anchor;
• score below 0.35 terminates advancement.

An intermediate anchor can contain two blocks only if its first block authorized a repeat. Advancement after that repeat must be justified by the second block's score. This closes a subtle loophole: validating only the last block at each anchor could accept a repeat that the first block never authorized.

Below the budget boundary, a high-scoring final block is not terminal. It is a promise to run the registered higher anchor. The result must refuse to exist until that promise is satisfied or a registered stopping rule applies.

The two budget boundaries do different jobs

The laboratory distinguishes an authenticated launch cutoff from an external provider hard stop. Exact credential names, balances, and operational thresholds are excluded. Budget-limited terminality still requires a fresh authenticated snapshot, zero live suites, and evidence newer than every cohort member.

Close the races, then publish once

Validation takes time. A suite could exit or a tracker could append while the freeze is hashing files. Passing the first liveness check is therefore not enough.

Immediately before publication, the gate repeats both Screen and process checks. It then rehashes the ledger and append-only status history, rediscovers the exact run-directory set, and recollects every file record. Any difference blocks the write.

The output itself is written to a temporary file in the destination directory, flushed and synchronized, then published with an exclusive hard link. The exclusive link fails if the target appeared concurrently. Readers see either no freeze or the complete freeze; they do not see a partially written JSON document. The command also refuses to overwrite an existing freeze.

This gives the result one birth event. Corrections require a separately named, versioned artifact rather than quietly changing history.

A freeze is useless if derivatives can walk around it

The rating, efficiency, and strength-efficiency commands treat the freeze as an active security boundary, not a decorative citation.

Before fitting a terminal DeepSeek rating, the rating command reopens the freeze, verifies its manifest-content hash and content-set hash, checks every source member's current byte count and SHA-256, reconstructs every block from raw games, and replays terminal ladder validation. Its report binds both the freeze-file hash and the cohort content-set hash.

The efficiency report additionally requires its ledger run inventory to equal the frozen run inventory and refuses any active run. The strength-efficiency frontier requires the canonical DeepSeek game ledger to contain every frozen game path exactly once—no missing, duplicate, or extra DeepSeek row—and requires the rating's two freeze bindings to match.

That chain is:

raw bytes
  -> exact cohort freeze
      -> freeze-verified rating
      -> freeze-verified efficiency
      -> exact-game, exact-rating frontier
          -> publication bundle

The current publication bundle manifest preserves the boundary by treating live derivatives as non-publication evidence and requiring the DeepSeek freeze, terminal rating, and billing attribution for publication. A green draft inventory is not permission to promote provisional numbers. Publication mode is supposed to fail while terminal evidence is absent.

What the freeze proves

If the gate succeeds, it proves a narrow but valuable set of statements:

1. No managed DeepSeek Screen or detected matching harness process existed at both runtime checks.
2. The ledger and filesystem named the same unique set of canonical run directories.
3. Every included byte was inventoried and content-addressed, with no detected known credential marker or non-redacted credential field.
4. Raw game outcomes, provider errors, and token totals reconciled with process metadata, aggregates, and the ledger.
5. Every variant's observed anchor history obeyed the registered adaptive policy and ended by score, authorized repeat, or authenticated budget rule.
6. The ledger, status history, directories, and files did not change during validation.
7. The output was published once, completely, without overwriting an existing freeze.

Those are provenance and terminality claims. They make later statistical claims inspectable.

What it does not prove

The freeze does not prove that DeepSeek has any particular terminal Elo at the drafting boundary. There is no terminal E001 result in this post.

It does not prove that the exploratory all-Black, single-opening ladder generalizes to human chess, other engines, other time controls, or another provider deployment. It does not turn adaptive exploratory anchors into a preregistered confirmatory sample. It does not identify whether candidate assistance changes proposal quality, selection, search, value estimation, or tool compliance.

Historical run metadata recorded Stockfish 18 and its available configuration, but did not record the Stockfish binary checksum. The freeze preserves that limitation; hashing current game artifacts cannot recover a missing historical binary identity.

The content set also cannot prove that public chess positions were absent from model training. Known-marker secret scanning cannot prove the absence of every possible secret. Process-table checks cannot prove the absence of an unknown writer that leaves no recognized process signature.

Finally, token reconciliation is not billing attribution. Exact historical dollar claims require request-level provider records joined to run, game, ply, attempt, model route, and billed cost. An authenticated credential balance is a budget control, not a DeepSeek invoice.

Fail-closed design does not eliminate uncertainty. It stops us from laundering known uncertainty into a terminal-looking artifact.

Refusal is a research output

A pipeline that says “not yet” is doing scientific work.

The refusal records which obligation remains unsatisfied: an active process, a stale snapshot, an orphan directory, a raw/aggregate disagreement, an invalid repeat, an owed advancement, a credential leak, or a source race. Each failure is more informative than a plausible table produced from an ambiguous cohort.

The cultural lesson is simple: do not rely on the analyst to remember that a result is provisional. Encode provisionality into the artifact graph. Make the terminal file impossible to create early. Make downstream tools verify, rather than merely cite, their frozen source. Make publication fail for the right reason.

Then, when the result finally exists, its existence means something.

---

Audited implementation boundary

This post describes the following source revisions inspected at drafting:

• research/analysis/freeze_deepseek.py — 83725cd29c08ca1868a91d33df92cf4f4ed17fc06809d39760bd52e0fceff8f2
• research/operations/ladder_planner.py — 5cd817d96d824070b7859d52a2ad92fc89c974815c35cf81eb7229ee6d70094c
• research/release/DEEPSEEK_TERMINAL_RUNBOOK.md — 2cb43312bfb146fc7382bb0dc481c942ce25a00b94fb57b7b8f8ad7f513217fd
• tests/test_freeze_deepseek.py — 8c267207f71b25172f7e4a05da3220021b4539fda08c072e11244f5125b52e7a

These hashes identify the audited implementation, not the future terminal cohort. The future cohort must carry its own manifest and content-set hashes.