Harness Engineering for Scientific Agents

abstract

The first edition of this paper argued that the harness engineering discipline growing up around coding agents would have to be rebuilt for agents that do science, and proposed a five-part framework for doing it. This edition tests that argument against five months of new results: a settled taxonomy for coding harnesses, automated harness search with evidence about which changes transfer, the first end-to-end benchmarks for autonomous research, a policy-level statement of the validation bottleneck from Google DeepMind, and the first formally gated LLM control of a physical quantum experiment.

The framework mostly held up. Verification has moved from one component among five to the organizing concern, and I’ve recut the verification stack along the line between computational and inferential checks. I name three things the first edition underweighted (certificates, pruning the harness, and the fact that prose doesn’t transfer), update the open problems, and include notes from running a certificate-gated harness on a ten-problem research program in fault-tolerant photonic quantum computing.

Why a second edition

I wrote the first version of this in March,²⁹ about six weeks after the term “harness engineering” appeared. There were three primary sources then, all about coding agents, and I was arguing by extrapolation that scientific agents would need something structurally different. Five months later most of that extrapolation has been tested, some of it by people with far more resources than I have, and I’ve run a certificate-gated harness of my own long enough to have opinions grounded in something other than analogy.

The core claim held, and so did the five-component framework, though I’d now cut one component differently. Three things I underweighted turned out to matter a lot. The field also produced numbers I didn’t have in March, and they change the tone of the argument: the benchmarks are in, and they don’t flatter anyone.

This edition is meant to be read on its own. Where I refer back to the March version, I say so.

1. The state of the coding-agent harness

The vocabulary settled faster than I expected. Birgitta Böckeler’s April piece on martinfowler.com gave the field a working taxonomy, borrowed from control theory.⁷ A harness is made of guides, which are feedforward controls that shape what the agent does before it acts, and sensors, which are feedback controls that observe what it did and push a correction back. Each comes in two execution types: computational (deterministic and fast, like a linter or a structural test) and inferential (an LLM reading the output and forming an opinion). An agent with only feedforward controls follows rules and never learns whether they worked, and one with only feedback keeps making the same mistake until a sensor catches it, so you need both. The interesting design questions are where each one sits in the loop and what it costs to run.

I’m adopting that vocabulary because it’s better than what I had. My March verification stack was ordered by cost. That’s a reasonable axis, but the split between computational and inferential checks matters more, because it tells you which checks you can trust unconditionally.

The second development is automated search over harnesses. Lin et al. (arXiv 2604.25850, April)⁹ closed a loop in which observability data from agent trajectories drives automatic mutation of the harness around a frozen model. The evolved harness transfers across model families and gains 5 to 10 percentage points on Terminal-Bench 2. HarnessBank (arXiv 2607.13683, July)¹⁰ does something similar with a gene bank and a screening gate, so it doesn’t have to evaluate every offspring, and reports gains of 5.1 to 15.4 percent across seven benchmarks. Bilevel Autoresearch (arXiv 2603.23420)¹⁴ puts a meta-loop over Karpathy’s autoresearch loop and reports a fivefold improvement on the same task.

For what follows, the most important result in that line of work is an ablation in the Lin paper. The gains came from tools, middleware, and long-term memory, with little or nothing from the system prompt, and only the structural changes transferred across models. So engineering effort on a harness should go into things the model can call.

The third development corrects something I got wrong by omission. In April, Anthropic published a short design note⁵ with a pattern I hadn’t considered: take harness assumptions out as the model improves. A harness encodes beliefs about what the model can’t do on its own, and those beliefs go stale. Their example is context resets that had been added to make up for a weaker model and had become dead weight by the time a stronger one shipped. The popular framing of harness engineering, from Hashimoto¹ and later Osmani, is a ratchet: every mistake becomes a permanent constraint, and the harness only tightens. That’s right as far as it goes. But a harness that only tightens will eventually be doing work the model no longer needs, and doing it badly, since constraints written for a weaker model can hold back a stronger one.

A cautionary case in the other direction landed the same month. Anthropic traced a visible regression in Claude Code to three harness changes:⁶ a lower default reasoning effort, a caching bug that dropped thinking history from stale sessions, and a prompt meant to limit verbosity that went too far. The model itself hadn’t changed. Each of those was a harness change made in good faith, and together they were enough to degrade the product in a way users noticed.

So in five months the coding-agent discipline has picked up a taxonomy, automated search, evidence about what transfers, and an explicit tension between ratcheting and pruning, all of it still about coding agents.

2. What the scientific-agent literature did in the same period

2.1 Autoresearch and the minimal loop

Karpathy released autoresearch on 7 March.¹⁵ It’s about 630 lines across three files. The agent may edit one file, train.py, a single-GPU pretraining script derived from nanochat, and may not touch prepare.py, which scores the result. The loop is to read the code, propose a change, train for five minutes, keep the change if validation loss improved and git-revert it if not, then repeat. Karpathy’s own two-day run produced about 700 experiments and stacked up 20 additive improvements. Within a month the repo had 66,000 stars.

I bring it up because it’s about as small as a scientific harness gets, so every problem shows up in miniature. Each keep-or-revert decision rests on a single noisy run, and nothing is repeated. Some improvements didn’t reproduce across sessions. The claim that gains at depth 12 carry over to depth 24 is a hypothesis, and a “keep” can be noise. Karpathy knows all this and says so. Still, the community’s most popular scientific harness has no statistical check at all. It has a metric and a revert, which gets real work done but will also let you stack false improvements on top of each other.

2.2 The benchmarks arrived

In March I had no numbers. Now there are three sets, and they agree with each other.

ResearchClawBench (arXiv 2606.07591, June)¹⁷ has 40 tasks across 10 domains, each derived from a real paper that’s hidden from the agent. The agent gets the task description, related literature, raw data, and an executable environment, and has to produce the research. A score of 50 means it reproduced the target paper. The strongest autonomous agent, Claude Code, averages 21.5, and the strongest single model run through the benchmark’s lightweight harness, Opus 4.7, averages 20.7. Failures cluster in three places: mismatched experimental protocol, mismatched evidence, and what the authors call a missing scientific core. One of their case studies is a quantum characterization task where the agent stopped at the most obvious observable trend and never built the gate-counting fidelity model or ran the mirror-circuit inference the target paper needed. The execution was fine, but the agent didn’t know what a physicist would consider done.

MLReplicate (arXiv 2605.16616, May)¹⁸ asked six autonomous research systems to reproduce eight ICML 2025 outstanding papers, then put the results through automated conference-style review and after that through human experts. The human reviewers found fabricated or unsupported claims in 59 percent of the papers the automated review had accepted. Automated and human overall ratings correlated at r = 0.29. And the cheapest system beat the most expensive one on human evaluation, despite a 38-fold difference in input tokens.

The verification-gap survey (arXiv 2608.05179, August)¹⁹ audited the runnable autonomous-research systems in the literature. Thirty-eight percent release the seeds or traces needed to reproduce a run, and 38 percent report any method for verifying novelty. Of the nine systems that claim closed-loop autonomy, seven are mechanical re-runs and one is author-asserted with no external check. Under the survey’s coding rule, no LLM-era system demonstrates an externally validated oracle inside the loop; the one system that does predates LLM agents.

Read together, the three papers say that these systems can execute a research workflow but can’t yet tell whether the result was science, and that the automated judges we’ve been using to score them can’t tell either.

2.3 The validation bottleneck, named

In July DeepMind published a policy essay called “Conjecture Machines.”²¹ It’s written for funders more than engineers, but it’s the sharpest statement I’ve seen of the problem I was circling in March. Agents are conjecture machines: they make ideas and candidate solutions cheap and plentiful. Refutation is still physical and institutional, which means slow and expensive. Popper’s pair has become lopsided in cost, and in most disciplines the validation gap is widening. Thang Luong, who led the Aletheia effort, calls it “proof indigestion”: AI produces results faster than people can review them.

A coding-agent paper from June says the same about its own domain. “The Verification Horizon” (arXiv 2606.26300)¹³ argues that the classical asymmetry, where checking a solution is easier than finding one, is reversing for coding agents. Generation has become the easy part and reliable verification the hard part, and the authors don’t think any single mechanism will fix that.

I think this is the most important fact about scientific agents in 2026, and it changes the framework. In March I listed verification as one component among five, which undersold it; I now think the other four exist to serve it.

2.4 Mathematics as the accelerated testbed

I want to state the May math results carefully, because they show what verification looks like when the domain cooperates. On 20 May, OpenAI reported that an internal model had produced the key ideas that disprove the Erdős unit distance conjecture, an open problem in discrete geometry, and that a group of mathematicians then checked, refined, and published the argument.²² Separately, DeepMind’s Aletheia solved six of the ten problems in the First Proof challenge, a set of unpublished research problems held out of training data. Aletheia pairs a proof generator with a natural-language verifier that sends flaws back for revision.

That makes two verification regimes: natural-language reasoning checked by people, which scales only as fast as people can review it, and proofs written in Lean and confirmed by a machine, which scale with compute. DeepMind’s framing is that math is the accelerated testbed for the rest of science, because it’s the one field where refutation can be made as cheap as conjecture, and the tools built there (certificate-checked artifacts, generators paired with verifiers, structured handoffs between the two) are tools every other field will need in adapted form.

I agree, and I should have said so in March. I underweighted certificates.

2.5 The physical world

This was the part of the March paper that was pure extrapolation, and it’s where the most concrete progress happened.

A group at Duke (arXiv 2606.27231, June)²³ put an LLM agent in the loop on a trapped-ion experiment with a formal safety boundary around every operation. The agent writes native ARTIQ control code. It reaches the hardware only through an MCP server, and no tool call executes unless it carries a single-use authorization token bound to the exact contents of that call. Tokens are issued in two ways. The automatic path runs the proposed script in an isolated simulation of the hardware (dax.sim) and checks every operation against preset per-device bounds; the manual path is a person, for actions classed as sensitive. An AST checker enforces a denylist before anything runs. They validated the filter with about 1,932 bench scripts and 250 harness-level tests, mapped where its protection ends, and deployed it on co-trapped Ca+/CaOH+ and Yb+ crystals.

That’s the authority model from my March paper, built, with one refinement I hadn’t specified: the automatic path to authorization runs the action in simulation first, which checks capability, policy, and competence in one pass. Section 3.2 says why I think that matters.

A-Lab GPSS (arXiv 2604.11957, April)²⁵ ran a 352-sample campaign on air-sensitive lithium halide spinels, with LLM agents split into an abductive role (spot the anomaly) and an inductive role (find the pattern). The hit rate for materials with both good ionic conductivity and high spinel purity rose from 1.33 percent in the first 75 samples to 5.33 percent in the last 75. The authors’ limitations section is the interesting part. The agents kept invoking cation ordering to explain differences in conductivity, and the lab had no characterization tool that could confirm it or rule it out. The hypothesis couldn’t be falsified inside the loop, so it stayed a hypothesis and kept steering experiments. The two next steps the authors name are more characterization modalities and calibrated uncertainty, so the agent can defer to a person when it doesn’t know.

And a July paper (arXiv 2607.04508)²⁶ builds an agentic self-driving lab explicitly around the validation bottleneck. Their framing is that the rate-limiter has moved: for high-value, low-throughput science, the physical experiment now sets the schedule, far more than the agent’s turns do. Their mechanism is an uncertainty-gated agent that trusts a cheap surrogate prediction only when its calibrated uncertainty is low and triggers a real measurement otherwise. It’s the multi-fidelity promotion I described in March, with the promotion criterion made explicit.

3. The framework, revised

The March paper proposed five components: a knowledge mesh, an authority model, a verification stack, an experimental planner, and a provenance layer. I’ll take them in order.

3.1 Knowledge mesh

I argued that the repository can’t be the single source of truth for a scientific agent, because scientific knowledge lives in instruments, databases, literature, and the heads of the people who run the lab, and that the replacement should be a queryable graph the agent pulls from instead of a document it’s handed.

The field’s partial answer has been agent skills. DeepMind’s science-skills repository, and the skills convention more broadly, let a scientist write down a piece of methodological craft as a text file the agent can load. In the DeepMind essay,²¹ Latysheva describes distilling parts of her research process into skills and expects research to shift from execution to orchestration. Skills do move some tacit knowledge out of people’s heads and into files.

But skills are prose, and the Lin ablation says prose is the part of a harness that doesn’t transfer. I take that as support for the March position. A skill that says “when the Ramsey fringe looks like this, the qubit has drifted” is a guide. What the agent needs at the moment of decision is a sensor: a fitted model it can call, which reports drift with an uncertainty. The knowledge mesh should be mostly callables with typed outputs, and documents only secondarily. Where a piece of knowledge can’t be made callable, it should still be written down, with the understanding that it’s the weakest layer of the mesh.

3.2 Authority model

The March version defined effective authority as the intersection of what the agent can do (capability), what it may do (policy), and what it has shown it can do (competence). The Duke system implements that model, and the detail I now think matters most is that its automatic path to authorization is a rehearsal in simulation. Only a clean simulated trace earns a token, and the token covers exactly the call that was rehearsed, so the agent can’t get authorization for one thing and then do another.

The sim-gate checks policy and competence with one mechanism. Policy is the bounds, and competence is whether the script, run against the model, does something sensible. If every action is rehearsed before it’s performed, you don’t need a separate certification eval for competence. You do need a good enough simulator, which in quantum control isn’t a given, and you need to know where the simulator is wrong, which is the same problem as knowing where the agent’s world model is wrong.

The manual path for sensitive actions is the escalation protocol. Their design classifies actions up front instead of trusting the agent to know when to ask, and I agree with that choice: an agent’s judgment about whether an action is dangerous is exactly the judgment you’re trying not to rely on.

I’d add one thing from experience. Authority should decay. A competence certification earned against last month’s device state should count for less this month. Physical systems drift, and a token-issuing simulator that isn’t re-fitted to the current device is issuing tokens based on a device model that’s out of date.

3.3 Verification stack

This is the component I’d cut differently. The March version had five layers ordered by cost and numbered 0 through 4: syntactic, physical, statistical, contextual, and adversarial. I still think the layers are right, but I’d now split them along Böckeler’s line instead of by cost.

Computational verification is anything a deterministic program can decide: units, shapes, bounds, conservation laws, symmetries, and whether a claimed uncertainty is justified by the sample. That covers my old layers 0 through 2. These checks can be trusted unconditionally, they’re cheap, and every output should pass through all of them. When a domain admits a certificate, meaning an artifact a checker can validate without trusting whatever produced it, the certificate goes here too. A Lean proof is a certificate. So is a stabilizer tableau that reproduces a claimed fidelity, or a syndrome history that decodes to the claimed logical error rate.

Inferential verification is anything that needs a model to form an opinion: is the result consistent with prior work, is the discrepancy explained, would a skeptical reviewer accept the argument. That’s my old layers 3 and 4. These checks are expensive and can’t be trusted unconditionally, and MLReplicate’s r = 0.29 between automated and human review is the best current estimate of how far to trust them, which isn’t far. Its finding that 59 percent of the papers the automated review accepted had fabricated or unsupported claims shows what happens when inferential verification is the final check.

The engineering consequence is that the job is to push as much of the stack as possible into the computational band. Certificates do that, and it’s why math is ahead. In March I treated the tension between discovery and drift as an epistemic problem, and part of it is, but a lot of it comes from domains that haven’t built a certificate format yet, and building one is engineering work.

The tension DeepMind describes, between agents tuned for caution that drift toward the known and agents tuned for originality that go astray, is the discovery-versus-drift problem from March in different words. I think cheap refutation resolves it, because then an agent tuned for originality can afford to be wrong often. Where refutation is expensive, the tuning question is real and there’s no good answer yet.

3.4 Experimental planner

The March version gave the planner four responsibilities: cost-aware selection, multi-fidelity routing, batch design, and abort-and-pivot. Routing now has an explicit promotion criterion, from the validation-bottleneck paper: trust the current level when its calibrated uncertainty is low enough, and move up to the next fidelity otherwise. That’s cleaner than what I wrote.

I’d reframe abort-and-pivot entirely. Sibyl-AutoResearch (arXiv 2605.22343, May)¹⁶ diagnoses where autonomous research systems lose the lessons of their own trials: weak evidence turns into prose, pilot signals turn into broad claims, memory stays textual, and recurring process failures don’t change later behavior. Their fix is two auditable conversion units. Trial-to-behavior conversion links a signal from one trial to a later research action, so that a failed experiment demonstrably changes the next plan. Trial-to-harness-behavior conversion links a recurring process failure to a change in the harness itself. In a strong harness, they say, past trials change future research behavior and recurring failures change the harness. That’s agent-harness co-evolution, and it describes the planner’s job better than “know when to pivot” does. The planner is where trial history becomes the next action, and that conversion has to be inspectable, or you can’t tell whether it happened.

The rate-limiter observation belongs here too. If the physical experiment dominates the schedule, agent turns are nearly free compared with measurements, and the planner should spend them: run the simulation ten more times, refit the surrogate, have the adversarial check argue with itself.

3.5 Provenance

In March I wrote that provenance is a requirement of scientific integrity. According to the August survey, 38 percent of runnable systems release enough to reproduce a run.¹⁹ So most of them don’t meet the requirement.

Two concrete mechanisms have appeared. AutoResearchClaw (arXiv 2605.20025)²⁰ keeps a registry of verified results and blocks numerical claims that aren’t grounded in it from the strict sections of a generated paper, and it runs a citation-verification pipeline that removes references it can’t resolve. Provenance works as a gate there, which I think is right: a claim that can’t be traced to a registered result doesn’t get written. DeepMind proposes Human-AI Interaction Cards,²¹ short records of the prompts and outputs behind the key insight in a piece of work, as an artifact for reviewers.

The uncertainty budget I asked for in March is still missing from essentially every system I’ve looked at. The A-Lab GPSS authors list calibrated uncertainty as a next step, and the validation-bottleneck paper builds its whole promotion mechanism on it. It’s the piece of provenance that would change agent behavior most if it existed, because an agent that has to carry an uncertainty budget forward can’t quietly turn a pilot signal into a broad claim. The budget wouldn’t close.

4. Notes from running one

Empiricist is a certificate-gated generate-verify-refine harness that I’ve been running since spring against a program of ten problems in fault-tolerant photonic quantum computing. As of this writing, two of the ten are fully certified. I keep results Empiricist produced strictly separate from results I got by hand, and I say which is which every time, because keeping them apart is the provenance discipline I’m arguing for.

A few things surprised me.

The certificate format is most of the work. Deciding what artifact a checker can validate without trusting the generator is a design problem in its own right, and it’s domain-specific in a way that resists templates. For a fusion-network problem, the certificate might be a stabilizer computation the checker re-runs; for a threshold claim, it’s a decoder run on held-out syndromes with a stated confidence interval. Getting the format right took longer than building the loop, and the two problems that certified are the two where the format was cleanest.

Refinement is where drift happens. Generation produces candidates, verification rejects most of them, and the refine step takes the rejection and tries again. If the rejection is prose, the refine step reasons about the prose, and it can talk itself into believing the checker was wrong. If the rejection is structured (which constraint, which value, off by how much), there’s less room to argue. Sibyl’s “weak evidence becomes prose” is the same failure running in the other direction.

The agent’s confidence carries no information. The cases where it was surest weren’t the ones that certified. About halfway through I stopped reading its confidence statements and read only the checker output, which is what the computational band is for.

And the sim-gate is cheap insurance. Everything Empiricist proposes runs against a model before it runs against anything expensive. I didn’t get this from the Duke paper. I did it because I didn’t want to burn GPU hours on candidates a stabilizer simulator could reject in milliseconds, and the Duke group did it because they didn’t want to burn a trap.

5. What I got wrong in March

Certificates. I treated verification as an epistemic problem with a mechanical front end. It’s closer to the reverse: an engineering problem of extending the certificate frontier, with an epistemic residue that shrinks as the frontier moves. The math results are the existence proof.

The ratchet. I described the harness as accumulating constraints monotonically, and a harness that only tightens becomes a ceiling. The better rule is that every constraint carries a record of the failure that justified it, so that when the model stops making that mistake, the constraint can come out with a clear conscience. The Claude Code regression shows what harness changes look like without that record.

Prose. I described my knowledge mesh as nodes and edges, which sounds structural, but most of my examples (protocols, papers, calibration histories) were text. The Lin result says text is the part that doesn’t transfer. The mesh should be callable.

And one thing I got right for the wrong reason. I said the split ontology, with knowledge in one medium and physics in another, was the hard problem. Building a bridge between the two turned out to be tractable; as Duke showed, it can be a simulator and an MCP server. The hard part, which nobody’s harness does yet, is re-fitting that bridge continuously to a physical system that drifts.

6. Open problems, updated

Continuous re-fitting of the authorization simulator. The sim-gate is only as safe as the simulator is current, and nobody I’ve read re-fits the gate’s world model against live device data as part of the loop. It’s the first thing I’d build next.

Certificate formats for domains that lack natural ones. Wet-lab biology has no Lean. What artifact could a checker validate for a claim about a cell line? The A-Lab GPSS case, where an unfalsifiable hypothesis kept steering experiments because the lab lacked the instrument to test it, shows the failure mode: the certificate format was missing, and the loop kept running anyway. A harness should know when a claim can’t be certified in the current lab and refuse to spend bench time on it.

Automated judges. MLReplicate’s r = 0.29 is the number to beat. Until inferential verifiers correlate with expert judgment, they’re sensors that report noise, and a harness that trusts them will wave through the kind of fabricated and unsupported claims MLReplicate found.

Harness search for scientific loops. All of the automated harness-evolution results come from coding benchmarks with fast oracles. Nobody has run Meta-Harness or HarnessBank against a loop whose oracle is a six-hour measurement. I expect the search will have to run mostly against simulated oracles and be checked against the real one only occasionally, which is the multi-fidelity problem again, one level up.

Decaying authority. Competence should expire, and no system I know of lets it.

Tuning for novelty versus caution. Still open. My position is that it’s a downstream symptom of expensive refutation and should be attacked there.

7. Where things stand

In March I ended with a metaphor about postdocs. I’ll skip it this time.

Coding agents now have a discipline, with a taxonomy, tooling, automated search, and a growing record of what transfers. Scientific agents have demonstrations, benchmarks showing those demonstrations aren’t science yet, and a clear statement from a major lab that validation is the constraint. So the harness work that matters for science is verification work: pushing checks into the computational band, building certificate formats, gating physical actions on rehearsal, and blocking claims that can’t be traced to a registered result.

All of it can be done now, with the models we already have, and the Duke rig and the A-Lab campaign show people doing it. Systems built this way will look slow in a demo, since most of their turns go to checking. I think that’s the right way to spend them.