The Bytes We Could Prove

A local request can be perfectly controlled and still become a different experiment somewhere between the compiler and the model

A105 began with an apparently simple promise: change whether a model can read a board panel or a complete legal-action panel, and change nothing else.

The first preflight made that promise executable. Across 128 locally rendered cells, the four factorial arms had equal request length, identical task and FEN, the same action universe and response schema, and differences only inside registered board and legal-content byte spans. The second preflight froze the analysis before outcomes existed.

That still left two uncomfortable questions.

First: is a repeated x really an inert mask?

Second: are the bytes emitted by the request compiler the bytes that cross the system boundary?

Neither question can be answered by looking at the pretty prompt. Both live in the machinery around it. So we added two more provider-free gates: A105-R0S, the opaque-mask sensitivity audit, and A105-R2, the capture-only transport fidelity audit.

They passed. Their more important contribution is showing exactly what a pass does not prove.

Equal bytes are not equal tokens

The registered renderer hides each board square or legal action with a run of x bytes of the same UTF-8 length. This gives us an exact local invariant: a four-byte UCI move becomes four mask bytes; a one-byte square symbol becomes one mask byte. Labels, delimiters, slot count, token lengths, panel order, task, full FEN, tool schema, and action universe remain unchanged.

But byte equality does not imply token equality.

xxxx, ####, !?!?, and a deterministic punctuation digest can be split differently by a tokenizer. They can also differ in visual texture, run structure, familiarity, and attention. If one mask is easier to compress or more salient, the mask itself has become part of the treatment.

A105-R0S does not wave this away with a proxy tokenizer. It freezes four mask families:

Family	Alphabet	Distinct bytes	Maximum run	Runs
mono_x	x	1	5	5,132
mono_hash	#	1	5	5,132
alternating_punct	!?	2	1	8,304
digest_punct	11 punctuation bytes	11	3	8,072

Each family masks exactly 8,304 hidden bytes across the fixture set. The audit renders 512 cells—16 fixtures, four treatment conditions, two panel orders, and four mask families—and groups them into 128 matched sensitivity blocks.

Every family preserves request length, registered spans, visible content, canonical envelope, hidden-token count, hidden-token byte-length histogram, schema validity, and deterministic rerendering. All 32 all-visible blocks are identical across families, as they should be: when both panels are revealed, there is nothing left to mask.

The audit deliberately reports the mask statistics instead of pretending they are equal. It also reports zero tokenizer counts.

That zero is a feature. We do not have exact frozen offline tokenizer artifacts for openai/gpt-5.6-luna or deepseek/deepseek-v4-flash. A guessed alias or a convenient third-party encoding would manufacture precision at the wrong boundary. The registered conclusion is narrower: all four masks satisfy the local byte-and-span contract. Their exact tokenization remains part of the future realized treatment and must come from model/provider telemetry.

No mask family was selected from outcomes. The original mono_x compiler remains primary until a separately frozen design says otherwise.

A canonical request is not a wire request

The second gap appears one layer later.

The chess harness builds model configuration, hands messages to an agent framework, and ultimately relies on client libraries, routers, proxies, and provider defaults. The local game logs do not expose a stable callback containing the final outbound HTTP body. Therefore we cannot infer provider-received bytes from the canonical request hash.

A105-R2 inserts a deliberately limited adapter in front of an in-memory capture sink. It never receives a credential or endpoint authority. It has no network socket, retry loop, response method, or provider client. Its only job is to normalize the 128 canonical R0 cells into two frozen serialization profiles:

• an OpenAI-compatible chat-completions shape; and
• an OpenAI Responses shape.

This produces 256 captured requests and 64 matched factorial transport blocks. For every fixture, order, and profile, the four arms must agree on route, header names, tool contract, parameter contract, normalization profile, and body byte length. Relative to the masked control, complete canonical bodies may differ only at the shifted board and legal spans. Every recorded body hash must verify independently.

The adapter also freezes one subtle default: when high reasoning is present, the canonical temperature field is removed uniformly from every arm. A condition-specific default would be treatment leakage even if the visible prompt stayed perfect.

All registered failure counters are zero. The adversarial suite still proves that dropped or injected fields, arm-dependent route or headers, tool-schema drift, out-of-span body mutation, unregistered profiles, and dishonest body hashes are rejected.

Where the proof stops

The phrase “transport fidelity” is dangerous unless its boundary is named.

A105-R2 proves bytes delivered by the repository adapter to its in-memory sink. It does not prove what the agent framework later serializes. It does not prove what an HTTP library adds. It does not prove proxy or router transformations. It does not prove what a provider receives, retains, tokenizes, or executes.

The production files inspected around that boundary are hash-bound as provenance, but they were not modified to create a fictional observability hook. The future paid runner must either use the audited adapter directly or add a separate interception at the actual client boundary.

This distinction is the systems equivalent of separating synthetic analysis from model evidence. A local pass can authorize the next gate. It cannot borrow the claim of a gate it never crossed.

Four preflights, four different claims

The A105 instrument now has four layers:

Layer	What passed	What remains outside the claim
R0 renderer	128 cells, 32 byte-matched factorial blocks	final positions, provider behavior, model effect
R0S masks	512 cells, 128 sensitivity blocks	exact tokenization, salience, semantic equivalence
R1 analysis	256 blinded stipulated synthetic cells	empirical effect, power, model or chess evidence
R2 transport	256 captures, 64 profile-specific blocks	framework, HTTP, proxy, router, provider-received bytes

None made a provider call. None invoked a chess engine. None observed a Luna or DeepSeek outcome. None licenses a causal headline.

Together they do something more valuable than a premature result: they make the failure surface explicit. Mask choice, tokenization, serialization, defaults, inference, attrition, and multiplicity can no longer hide inside one sentence that says “the prompts were matched.”

The lesson for LLM experiments

In an ordinary software test, we often own both ends of the function. In an LLM experiment, the treatment passes through layers we may not own and cannot fully observe. The visible prompt is only one representation of the intervention.

A serious claim therefore needs a chain of custody:

1. the abstract treatment schedule;
2. the executable request compiler;
3. sensitivity to arbitrary but consequential representation choices;
4. normalization into each transport profile;
5. capture at the real client boundary;
6. provider/model telemetry;
7. the frozen outcome compiler; and
8. the public evidence bundle.

We have proved the first four locally. The fifth remains a real execution gate. The sixth and seventh await paid outcomes. The eighth must publish nulls, adverse effects, attrition bounds, route and order sensitivities, and every registered failure—not only the most flattering contrast.

The bytes we could prove are now content-addressed.

The bytes we could not prove are now named.

That is a better foundation for the result than pretending the boundary does not exist.

Reproduce the provider-free gates

From the repository root:

uv run python research/experiments/run_a105_mask_sensitivity.py
uv run python research/experiments/run_a105_transport_preflight.py
uv run pytest -q tests/test_a105_mask_sensitivity.py \
  tests/test_a105_transport_fidelity_preflight.py

The generated reports are:

• research/analysis/a105_mask_sensitivity_v1/analysis.json
• research/analysis/a105_transport_preflight_v1/analysis.json

They contain local systems evidence only.