The Price of Thinking

Cheap tokens can still produce a slow, expensive-to-operate agent

A model can be cheap on a price sheet and still be impractical inside an agent.

DeepSeek V4 Flash made that distinction impossible to ignore. In our chess harness, as many as five concurrent evaluation suites stayed alive for more than a day. Provider retries occurred, but none became a persisted terminal provider-error game at the cited snapshot. The machine was not busy calculating Stockfish moves. Most of the elapsed worker time accumulated while waiting for long, high-reasoning model turns inside long, multi-turn games.

By the snapshot at 2026-08-04T21:10:33Z, the DeepSeek cohort had accumulated:

• 43 clean completed games;
• 24.43 million recorded tokens;
• 157.55 aggregate worker-hours;
• 41.61 hours of program wall time;
• 3.79 workers active on average; and
• zero recorded provider-error games.

Those numbers describe one system under one protocol. They are not a universal benchmark of the model, and the dollars cannot yet be attributed cleanly to the DeepSeek cohort. But they establish a systems result that matters beyond chess:

In an agent, the cost of thinking is measured in tokens, latency, sequential turns, and occupied workers—not tokens alone.

A zero-error ledger is narrower than it sounds

We subsequently made the missingness rule executable instead of leaving it as prose. A content-addressed point-in-time census at 2026-08-05T00:03:02Z covered 269 persisted Luna and DeepSeek game attempts under 29 suite manifests: 269 clean games, zero terminal ERROR OCCURED records, and 21 planned clean games not yet written. The aggregate W-D-L was 137-59-73, for a clean-conditional score of 0.61896. With no terminal error records, assigning provider errors as losses or draws leaves that number unchanged.

That does not establish perfect provider reliability. Frozen game/run JSON does not contain request-level retry counts, requests that never produced game JSON, returned route/model IDs, request IDs, or structured exception details. Nor does a suite's planned target prove that each unwritten game was attempted. The analysis therefore treats the 21-game shortfall as scheduling/right-censoring, reports it separately, and marks those fields unavailable. This is a generated_live nonterminal systems snapshot, not the final DeepSeek result.

One hour can mean five hours

Concurrency makes runtime language treacherous.

The first DeepSeek suite began at 2026-08-03T03:34:00Z. At the snapshot, 41.61 hours had elapsed since that launch. That is program wall time: what a person waiting for the research program experienced.

The suites did not all begin together. Regular play started first; top-10, top-5, top-3, and top-2 were added as evaluation slots became available. Their individual elapsed times sum to 157.55 hours. That is worker time.

The accounting identity is simple:

worker-hours = sum(elapsed hours for every suite)

five suites running for one wall-clock hour = five worker-hours

At this snapshot, 157.55 / 41.61 = 3.79, so the cohort averaged 3.79 active workers over its lifetime. Saying only “we ran DeepSeek for 41.6 hours” hides the parallel capacity consumed. Saying only “we ran it for 157.5 hours” sounds like nearly a week of human waiting. Both numbers are correct, but they answer different questions.

Quantity	Snapshot value	Meaning
Program wall time	41.61 h	Time since the first DeepSeek suite launched
Aggregate worker time	157.55 h	Sum of elapsed time across all DeepSeek suites
Average concurrency	3.79	Worker-hours divided by program wall time
Clean games	43	Completed result JSONs excluding provider-error games
Active suites	4	Suites still running at the snapshot

Worker-hours here are an orchestration metric, not GPU-hours. The API provider's hardware allocation is unknown. A waiting local process occupies an evaluation slot even when its CPU usage is negligible.

Why a chess game amplifies inference latency

A one-shot benchmark sends one request and waits for one answer. Our player is a tool-using agent. On every model move, it may inspect the board, request legal moves, reflect, correct an invalid action, and finally submit a move. A game can last dozens of model moves. The protocol permits up to 200 plies.

The latency stack is therefore multiplicative:

game latency
  = model moves per game
  × agent turns per move
  × latency per model turn

Candidate generation is not the dominant explanation. Stockfish receives only 0.1 seconds per move for the shortlist. The rated opponent also uses a short wall-clock search. The long spans arise from repeated remote, high-reasoning turns along the game trajectory.

This is also why concurrency helped less than intuition suggested. Five workers can hide latency across five games, but they cannot parallelize the sequential decisions inside one game. Move 42 cannot be requested until move 41 has changed the board.

The tokens are real; their interpretation needs care

The DeepSeek game records contain 2.13 million prompt tokens and 22.30 million completion-side tokens at this snapshot:

Recorded token class	Tokens	Share
Prompt	2,132,791	8.7%
Completion-side	22,298,354	91.3%
Total	24,431,145	100%

The adapter reports completion-side usage in a field named completion_tokens; under this provider/model path it includes the large reasoning-side output charged to the request. We therefore call these completion-side tokens, not a direct transcript of private reasoning.

Across 43 clean games, the arithmetic average is 568,166 recorded tokens per completed game. That ratio is descriptive, not a prediction for the next game. Game length, terminal reason, candidate condition, action retries, and current position all change the amount of inference.

The variation across suites is already large:

DeepSeek suite	Anchor	Status	Clean games	Tokens	Tokens / clean game
Regular	1320	Complete	10	7.60M	759,723
Top 10	1320	Active	8	4.89M	610,880
Top 5	1320	Active	6	5.55M	924,349
Top 3	1320	Active	8	2.95M	368,828
Top 2	1320	Complete	10	3.41M	340,621
Top 2	1720	Active	1	43,945	43,945

The last row is especially easy to misuse. One short completed game at a new anchor says almost nothing about that suite's eventual average. Active games have also consumed time—and may have consumed provider-side tokens—that are not present in the clean-game totals until their result artifacts are written.

Luna provides scale, not a controlled efficiency comparison

The completed Luna pilot contains 220 clean games and 25.29 million recorded tokens across all ladder blocks. That is about 114,953 tokens per game. DeepSeek had recorded almost the same total token volume—24.43 million—for only 43 clean games, about 4.9 times as many tokens per completed game.

Cohort	Status	Clean games	Prompt tokens	Completion-side tokens	Total tokens	Tokens / game
Luna	Completed pilot	220	7.53M	17.76M	25.29M	114,953
DeepSeek	Provisional snapshot	43	2.13M	22.30M	24.43M	568,166

This is not a clean claim that one model is 4.9 times less efficient. The cohorts differ in their realized games, terminal reasons, ladder anchors, provider implementations, and completion status. The DeepSeek denominator is still moving. The comparison is useful as operational scale: the second cohort consumed Luna-like token volume long before producing Luna-like sample volume.

The composition differs too. Completion-side tokens were 70.2% of Luna's recorded total and 91.3% of DeepSeek's. The excess is therefore concentrated on the generation/reasoning side rather than being explained only by larger board histories in the prompts.

Dollars are the least complete column

Billing totals, credential balances, and account-level budget state are operational data and are intentionally excluded from this public export. Token and runtime quantities remain because they are attached to experiment artifacts; no account-level amount supports a model-specific cost claim.

Cheap tokens and slow agents are compatible

Price, throughput, and latency are separate axes.

A model can have a low price per million tokens while producing many tokens per decision. It can have fast token generation while spending a long time before the first visible action. It can be inexpensive per game but too slow for an interactive product. It can be slow per game yet efficient in batch if enough parallel requests are available.

For the chess system, at least six quantities matter:

1. tokens per model turn;
2. model turns per move;
3. model moves per game;
4. latency per turn and per game;
5. worker-hours per completed game; and
6. billed dollars per completed game.

At this snapshot, 157.55 / 43 = 3.66 worker-hours had accrued per clean DeepSeek game, inclusive of time currently invested in unfinished runs. The cohort produced roughly 1.03 clean games per program wall hour because several workers overlapped. Neither number is an intrinsic speed benchmark. Both are useful capacity-planning measurements for this exact orchestration.

One frontier is closed; the replication is not

Efficiency without quality is not optimization. A fast system that loses every game is not efficient at playing chess.

Luna's ladders are terminal and can be placed on a completed-pilot strength-versus-token chart. Its top-2 harness is the sole nondominated point: approximately 2832 local Elo at 71,683 recorded tokens per clean game across 60 games. Every other Luna condition was both weaker and more token-intensive on that all-game arithmetic mean.

Figure — Luna candidate-count rating and token volume

That dominance is descriptive, not causal. A stronger harness may end games sooner, face a different ladder path, or avoid expensive recovery loops. The figure's token panel shows per-game medians, while the Pareto calculation uses the aggregate token total divided by all clean games. Both preserve the same top-2 conclusion.

DeepSeek's curve is still moving. At this snapshot:

• regular DeepSeek had completed its 1320 block at 0–2–8;
• top-2 had completed 1320 at 10–0–0 and begun a 1720 block;
• top-10, top-5, and top-3 remained incomplete at 1320; and
• no finite Elo should be assigned to the saturated 10–0 top-2 block.

The perfect lower-anchor score is a promotion signal, not an Elo estimate. The next anchor supplies the information needed to locate the system closer to a 50% score.

The analysis code computes a separately labeled provisional DeepSeek frontier so drift is visible during the run. It never pools that frontier with completed Luna. Top-2 currently leads the provisional set, but its fitted point depends on ten wins at 1320 and only one completed loss at 1720. That is a live diagnostic, not a publication result. We can publish Luna's completed frontier now; the cross-model frontier must wait for terminal DeepSeek evidence.

Better evaluation should spend information, not games

The first ladder used fixed ten-game anchor blocks. That is simple and easy to audit, but inefficient at saturated scores. A 10–0 block mostly says the opponent was too weak. Elo information is greatest when the score is near 50%.

Future ladders should treat inference as an experimental budget:

• promote early after a preregistered sequence of decisive wins;
• stop weak conditions early under a symmetric rule;
• allocate more games near the likely rating;
• interleave conditions to reduce provider-time drift;
• fit all anchors jointly rather than discarding lower-anchor evidence; and
• preserve the stopping rule so uncertainty accounts for adaptive sampling.

This is not merely a way to save money. It converts the same token budget into more statistical information.

The architecture can reduce cost at several layers

The obvious intervention is lower reasoning effort, but it is only one lever.

Compile the action interface

Return one legal move through constrained output instead of allowing repeated natural-language correction loops. This tests how much time belongs to chess reasoning versus protocol recovery.

Separate policy from explanation

Ask for a move first, then request analysis only when needed. Long prose should not be mandatory for every quiet position.

Trigger compute by uncertainty

Use shallow selection when one candidate dominates and deeper comparison only when values are close. An agent should spend tokens where the decision is actually ambiguous.

Cache deterministic state

Keep the board, legal actions, candidate analysis, and recent plan in a compact state layer instead of repeatedly reconstructing them through dialogue.

Narrow candidates progressively

A cheap policy can reduce ten moves to three; a stronger evaluator can compare the final two. The Luna result motivates this design, but does not yet prove it will preserve strength.

Distill expensive trajectories

Use high-reasoning assisted games as training data for a cheaper selector. The research system can be slow if the deployed policy learns not to be.

Every optimization must be measured against move regret and complete-game score. Cutting tokens by disabling the capability under study is not an efficiency gain.

What would change our mind?

The “reasoning verbosity drives latency” explanation weakens if request-level timing shows that most wall time occurs before generation, in provider queues, or in repeated network/tool overhead. It strengthens if completion-side token count predicts latency within matched positions after controlling game phase.

The candidate scaffold is not computationally efficient merely because top-2 uses fewer tokens per completed game in this snapshot. That difference may come from game length, outcome, or sampling. It becomes credible only if paired positions show lower tokens and latency at matched move quality, and repeated full games retain the strength advantage.

The case for adaptive compute weakens if fixed low reasoning preserves both selection regret and full-game strength. It strengthens if uncertainty-gated reasoning matches high-reasoning quality at substantially lower latency.

Measurements required next

The present ledger is game-level. The next efficiency study needs one row per model request:

• request start, first-token, and completion timestamps;
• provider and returned model identifier;
• prompt, cached, completion, and reasoning token fields;
• agent turn number, game ply, and action type;
• retries, invalid actions, and correction turns;
• candidate count and position difficulty;
• request-level billed cost or activity-export join key; and
• concurrent-request count at dispatch.

With those fields we can decompose:

latency per game
  = provider wait
  + generation time
  + tool time
  + invalid-action recovery
  + sequential dependency

Then strength, dollars, latency, and worker capacity can share one honest Pareto plot.

Thinking is a systems budget

The persisted DeepSeek result ledger was not terminally “broken.” That was the uncomfortable finding. The processes stayed alive, requests continued to return, games completed, and no provider-error game entered the ledger at the cited cut. This narrower claim does not rule out request-level retries or requests absent from persisted game JSON.

The specification was expensive in time.

Language-model evaluation often treats inference as a scalar token count added after the intellectual work is complete. For agents, inference is part of the intellectual design. A tool loop determines how often the model thinks. A state representation determines how much it must reread. A candidate set determines how many branches it explores. A stopping rule determines how long the experiment waits at an uninformative anchor.

The model is only one component of the player. The bill, the latency, and the occupied workers belong to the player too.

---

Snapshot and reproducibility note

Except for the explicitly dated later A103 missingness census, the operational snapshot values in this post are frozen to 2026-08-04T21:10:33.925405Z. The source-of-truth game and block indexes are experiment_tracking/games.csv and experiment_tracking/ladder_blocks.csv. The append-only account and active-screen chronology is experiment_tracking/status_history.jsonl. The reproducible worker-time report is generated by research/analysis/efficiency_snapshot.py; its snapshot semantics explicitly distinguish suite hours, worker-hours, and program wall hours.

Token totals count recorded game artifacts available at the snapshot. Active in-progress games can consume time and provider-side usage before those tokens enter the completed-game ledger. Dollar attribution remains pending an OpenRouter request-level activity reconciliation. The invalid early-key budget readings and their correction are preserved in experiment_tracking/corrections.jsonl rather than removed from history.