Linked methods note. The short Study page carries the interactive explanation; this page preserves the mechanism, motivation, controls, and open questions. Return to PDD in the study.
Cheap enough to test populations. PDD is a span-restricted fine-tuning method that inpaints rationale around a fixed target action, then applies loss only to accepted replacement words. The structured action schema is excluded from the target—but held-out tests must still show that syntax and behavior survived.
“We simulated a population of agents” often means one model prompted several ways: one policy wearing hats. PDD asks whether genuinely distinct, inexpensive policies can become an experimental variable.
Multi-principal safety assumes the other agents were built by somebody else: different training data, objectives, and blind spots. Their differences can interact when a commons fails.
A simulation where every agent shares a weight matrix instead measures prompt sensitivity. Two such agents can fail in correlated ways. The practical obstacle is arithmetic: thirty genuinely distinct fine-tuned agents used to cost more than the study.
For flockbench the requirement was odd: not better agents, but many different ones, each distinct in a way that survives a long rollout and each cheap enough that a population is a variable rather than a capital expense. And something most alignment training does not care about — they had to keep working.
Our environments parse structured actions out of free text. Before training, an agent may emit
{"reasoning": "the pool is low; take now to protect my balance", "action": "take"}
and the environment reads action. If training bends the model such that it stops emitting valid JSON, the run does not produce an interestingly-changed agent. It produces a parse failure, which our harness scores as a behaviourally inert action, and the experiment quietly measures nothing.
So the constraint was: try to change the agent's disposition without directly training on its schema.
Progressive Denoising Distillation (PDD) builds a target around a desired structured action. A diffusion-style teacher masks and inpaints selected reasoning spans; only accepted replacement tokens carry gradient, while the action field remains fixed and never enters the loss. This is not REINFORCE-style training: there is no scalar reward or policy-gradient update.
The crucial distinction is easy to lose in an abstract token diagram. The initial take below is a pre-training diagnostic. The restore action belongs to a separate target and stays fixed while its rationale is filled in. After distillation, a fresh rollout tests whether the model actually changed behavior.
Three things fall out.
The action schema is not a training target. This removes direct gradient pressure on the tokens that make the output parseable. It does not guarantee schema preservation: a changed model can still drift elsewhere, so held-out action-validity and parse checks remain mandatory.
Cheap, because the loss is sparse. Most of a rollout is unchanged, so training on the whole target spends most of the gradient re-learning text the model already produces. Constant factors are the entire game when you need thirty of something.
The teacher's edit is cached once. Every training arm consumes the same immutable cache, so no arm can get a fresher sample after results are known. A cost measure that turned into a fairness guarantee: the comparison is fair because re-rolling is impossible, not because we promised.
The longest Stage 0 run we tried—2,000 steps on Qwen2.5–7B—took about 3–4 GPU-hours. At the grant budget's planning rate of $2 per 80 GB GPU-hour, that is roughly $6–$8 for one training attempt. It took several attempts to reach useful convergence, so this is not a measured cost per finished adapter. The question under study is whether better recipes can make a usable, behaviorally distinct adapter cost considerably less than $50—and make the campaign, rather than the population, the expensive part.
The interesting quantity is not the price of one successful run but the ratio between what a population costs and what a campaign against it costs. One attempt has a measured timing; the number of attempts required for convergence is still a research variable. If that multiplier can be made small and reliable, population size becomes a dial and long evaluation rollouts can remain the expensive part.
That is the thing I actually want to explore: how far can the per-agent cost be pushed down before the agents stop being usefully distinct? There is a floor somewhere. Below some amount of training you have thirty copies of the base model with different accents, and everything downstream is measuring noise. Nobody has mapped where that floor is, and it is a cheap thing to map.
What the method is not: a better way to align an individual agent. The success criterion is not "the trained agent is more cooperative" but can we tell the seeded arm from a vocabulary-matched control at the population level? If not, it failed however good the individual agents look.
It is worth being explicit that the funder described this before we did. The Cooperative AI Foundation's call asks, under sandboxes and testbeds, about *"navigating the trade-off between scalability and fidelity, for example, by using smaller, distilled models to serve as faithful proxies for frontier agents in simulations."* That is the same sentence from the other side.
The failure mode that worries me most is not the obvious one.
Suppose we train thirty agents, run the population, and see no effect. Two explanations fit: the specification does not move population outcomes, or the training installed nothing and we bought thirty copies of the base model with different accents. Very different findings, identical in the data.
So: a positive control that costs nothing to check. Train toward something objectively verifiable and behaviourally irrelevant — write the reasoning in snake_case. Conformance is a regex.
It is much cheaper to test all of that with snake_case than with ethics.
I want to name these rather than wait to be asked.
We have not shown the method beats its alternatives. Three cheaper things come first — prompt specification, context distillation, and outcome-filtered self-distillation. Seed production is a comparative methods question, ordered cheapest-first, and reporting that the expensive method was unnecessary is a perfectly good result.
Distinct is not the same as distinct in the right way. Thirty agents differing in weights could still fail in correlated ways if the teacher imposed a single consistent bias on all of them. We do not yet have a good measure of population diversity as opposed to population size, and I think that is the real open problem behind this whole approach.
The cost figure is one number from one setup. Qwen2.5-7B, LoRA, our environments, our hardware, at one moment in a market where GPU prices move monthly. It is a real receipt rather than a projection, and it is not a law, a benchmark, or a promise about anybody else's stack. If the honest version of this work concludes that distinct-enough agents cost three hundred dollars each and you can therefore afford eight, that is a useful finding and I will publish it as one.
The honest reason this method exists is that I could not afford the alternative.
Every architectural decision above traces back to being one person paying for compute out of pocket. A judge panel is expensive, so we used arithmetic. Re-rolling the teacher is expensive, so we cached it once and got fairness for free. Training the whole target is expensive, so we masked the schema and got parse-safety for free. The constraint produced a discipline a better-funded version of this project would probably have skipped.
I would not claim poverty is a research methodology. But it is a fairly good filter for which parts of a pipeline were load-bearing.
*flockbench and Firebreak are CC0. The current grant narrative and interactive population results begin at the Commons Game overview. The style-control harness lives in continuous_judge; the population side, including the gate that scores it, is in flockbench.*