Archived research question 2 · linked design appendix. This page preserves the paired-cell design, estimand, power arithmetic, controls, and stopping rules from the superseded three-question framing. Return to the archived framing. Open the current Study.

How an experiment is run

What a cell is, what is paired with what, and whether the campaign can see the effect it was sized for.

The unit: one paired cell

A cell is one run of one configuration at one seed. Cells come in pairs: the same environment, the same seed, the same role mapping, with exactly one thing swapped. The measurement is the difference, not either number alone.

Pairing removes the seed; what is left is the swap. The pilot paired SD was 1,180.

What varies, and what is held still

factor	levels	why
Seeded fraction `f`	the sweep	The independent variable. What fraction of the population is yours.
Arm	baseline, candidate	The swap. At `f = 0` the two arms are the same population, which is what makes the A/A control free.
Environment	two, both core	The Shared Resource game, and a shared key-value store reached through auditable tool calls with memory carried between episodes. The second is core, not contingent: an answer key that only survives in a discrete-action game is the weaker claim.
Resource regime	zero slack, positive slack	Crossed with environment for the four strata the planner counts. Unwinnable runs alongside as a control, not a stratum — nothing should work there.
Seat provider	three sources	Two open-weight families served locally, plus a commercial model on some seats. Prompt and LoRA variants over one endpoint are within-source controls, not principals.
Model family	two	One dense, one mixture-of-experts. Reported separately, never averaged.
Salt	two levels	Which seats the seeded agents occupy. Guards against a result that is really about seat order.
Seed	many	Replication. The only thing that differs within a pair is nothing — the seed is shared.
Held fixed	horizon, N, prompt scaffold, sampling temperature, serving stack	Each is varied in its own study — horizon and N have a milestone of their own — rather than allowed to drift inside a campaign.

The estimand

At each fraction, a candidate is matched to f = 0 on seed, environment, slack, model source and seat order. f* is the smallest fraction that improves survival by the declared margin.

Primary fit is isotonic with bootstrap intervals. A non-monotone curve leaves f* undefined; raw curve published. The margin is the survival cost of one permanent defector among eight.

The campaign, and whether it is powered

Two numbers decide whether a campaign is worth running: what it costs, and the smallest effect it can see. Affordable and underpowered produces a null nobody can interpret. Both are arithmetic, so here they are.

The smallest effect this campaign could detect, against how many paired cells it buys. The dashed line is the margin you asked for.

Model families 2 Environment × regime strata 4 Seeded fractions 6 Salt levels 2 Seeds per configuration 35 Margin you care about 400 Paired SD 1180

Move the paired SD up and leave the margin alone. The requirement rises with the square of the ratio, so Phase 0 re-measures variance before the powered compute is committed.

The envelope is fixed and the scope is ordered. The core is gate calibration plus a powered dose-response across four environment × regime strata: 6,720 cells, 3,360 paired comparisons, 70 paired cells per contrast against the 69 the margin requires. Contingent, dropped in this order if Phase 0 needs the compute: N = 50, the unwinnable control, non-informative fraction points. 20,665 GPU-hours all in — $41,330 at $2.00 an 80 GB GPU-hour, plus $11,030 of commercial-API seats, so $52,360 of compute inside a $275,000 request. Gate calibration is 17% of that.

The break experiment: pointing the gate at a clone

The A/A is the cheapest important check in the plan.

The break experiment

The question	How often does the gate reject a candidate identical to the baseline?
Why it is free	At `f = 0` no agent occupies the seeded role, so both arms describe the same population, differing only in what an unused role points at. No new engine code, no new environment.
What is measured	Every difference between the arms is noise, so every rejection is a false one. The rejection rate is the false-rejection rate.
What it already found	Offline, not live. Sign-flip resampling of the 30-cell pilot rolled the overlap-rule gate back on a clone in approximately 100% of null resamples — and it worsens with more data, bounding a proportion that converges to 0.5 under the null. A superiority test fails oppositely: demanding significant improvement, a clone fails by construction. A provisional tolerated-harm threshold did better under the same resampling (0.076 at n = 30, 0.015 at n = 60), but is not a formal non-inferiority test.
Status	Not yet run live. Everything above is resampling of committed pilot data. Phase 0 runs thirty matched f = 0 campaigns of sixty pairs, spread across four strata rather than piled into one, to measure the live operating error. With no rollbacks the pooled exact 95% upper bound is 9.5% — looser than the 4.9% sixty single-stratum campaigns would buy, and deliberately so: a tight bound that holds only where it was measured is the less useful object. Above 10% the gate is declared unfit and we say so.
Pre-registered	The live rate will exceed the offline one, and the 400-unit margin will need re-deriving from measured variance. Written down before the campaign runs, so it can be wrong in public.
What it changes	Two artifacts ship with the gate that we are not aware of anyone publishing: a measured error rate, and a minimum campaign size below which its decisions are void, enforced by the validator.

A gate that rejects everything looks exactly like a careful gate until somebody points it at a clone.

Set the true effect to zero and watch three reasonable rules disagree about a clone, on the gate page.

The controls, and what each is against

Boundaries and stopping rules

This detail was removed from the short Study page so the reviewer path stays focused. It remains here because these tests determine how far a result can travel and when the project must stop making a claim.

boundary	what is tested rather than assumed
Cross-game transfer	The same seeded policy in both games. Stage 0 found opposite signs, so transfer is measured rather than assumed.
Tools and memory	A shared key-value store with auditable reads and writes, plus bounded cross-episode memory, while preserving the experimental invariant.
Multiple principals	Two open-weight families plus commercial seats. Prompt and adapter variants over one source remain within-source controls, not different principals.
Frontier fidelity	A matched frontier subset must place f* inside the open-weight interval and meet a declared action-rate tolerance before any proxy claim.
Scale	N = 8, 20 and 50 tests whether f* moves with population size. This is a boundary test, not a claim to have found a scaling law.
Safe execution	Sandbox-scoped credentials and simulated resources only; configurations, traces and receipts are released for replay.

Four ways the study says no. A flat curve means a controlled minority does not move survival by the declared margin. A non-monotone curve leaves f* undefined and the raw curve is published without repairing the estimand. A live gate error bound above 10% retires candidate decisions. A failed transfer is the boundary itself, and no universal threshold is claimed.

Pre-registration, and what a receipt contains

Pre-registration here is a versioned plan, not a promise. Before a scored campaign it states the cells, thresholds, estimand, direction of effect, and what would count as no effect. The runner freezes and compares resolved plan files. Commit provenance is recorded in release receipts; enforcing it in the runner is remaining work, not a property we claim.

Every gate decision emits an auditable JSON receipt — self-contained and versioned, not cryptographically signed — carrying the resolved policy, the paired deltas and the verdict. Our pilot's policy file changed after the run and the decision was still reconstructible, because the receipt embedded the policy it used. tools/replay_receipt.py re-derives every published figure from the receipt alone.

What could still make the answer wrong

threat	what we do about it
The threshold is not identifiable at N = 8	Reachable fractions are {0, 12.5, 25, 37.5}%, coarser than the interval of interest. Everything at N = 8 is a pilot. Resolution reaches 5 points at N = 20.
One serving endpoint	Today "different principals" means different specifications over identical weights — the honest state of Stage Zero. The funded core fills seats from two open-weight families and a commercial model, because relabelling prompts as principals is the thing to avoid.
Variance measured at the wrong scale	The case above. Phase 0 measures it first.
Small games may not transfer	Tested, not assumed: tool calls and persistent memory in the core, then N ∈ {20, 50}. If thresholds do not transfer, that is the finding.
Parse failures scored as behaviour	Recorded as an inert action and counted separately, never folded into the outcome. The last live sweep had zero.

The same arithmetic as flockbench's campaign planner, pinned by test. What has been run is on the status page.