Coding Agents Can Solve the Task — But How Much Work Did They Waste?

Here is a run that passed.Eleven turns. The tests passed on turn 4. Turns 5 through 10 were work nobody asked for, and some of it ended up in the final patch.The agent changed eight lines across three files. The reference fix is just two lines in one file. So six of those eight changed lines were unnecessary when compared with the reference.A normal pass/fail check would treat this run the same as a clean five-turn run. Both solved the task. There is no difference in the score.There’s one important thing I should say before going further: the agent in this experiment is simulated. I didn’t run Claude Code, Codex, or another real coding agent for this article. The repositories, edits, diffs, test runs, and failures are real. What I simulated was the agent’s decision-making, using a seeded Python policy. I’ll explain why I chose that approach, what it lets me measure, and where the limits are.The question I wanted to answer is pretty simple.When an AI coding agent takes extra turns to get to a passing patch, how much of that work was actually needed? And can a workflow around the agent reduce the unnecessary work?I ran about 50,000 simulated runs: 24,000 in a compliance sweep, 14,400 testing individual rules, and another 12,000 on a held-out suite.Three results stood out to me. And one prediction I made before running the experiment turned out to be wrong.Why a passing patch does not tell you how well an AI coding agent workedSWE-bench starts with a real GitHub issue and a real codebase. The model has to change the code and get the relevant tests passing. The benchmark has 2,294 problems taken from 12 Python repositories [1]. It is a tough benchmark, and the main score is easy to understand: did the patch fix the issue or not?That score is useful, but there is something it doesn't show. It only looks at the final result. It doesn't care how many steps the model took to get there.Say one run finds the failing test, tracks down the bug, makes the change and gets a green test run. Another run gets there too, but first changes the wrong file, backs that change out, runs the same command a few more times, changes some unrelated code and then runs tests that were already passing.Both runs are successful in SWE-bench. They don't look very similar to me, though.The second run used more turns. It took more time and tokens, and there is more for someone to look through. If some of those changes survive into the final patch, the extra work shows up there too.For this experiment, I wanted to keep two things separate:Final patch size: what is different between the starting repository and the final patch.Total work: everything the agent did before getting there, including changes it eventually reverted.What a pass rate sees, and the work it leaves out.A pass rate scores only the final patch. Below that line sits everything an AI coding agent did to get there: failed test runs, repeated commands, edits to the wrong file, reverted changes and passing tests run again for no reason.The rest of this article looks at what happens before that final patch.How I built a coding agent benchmark without running a real agentI had two restrictions from the start: no API keys and no downloaded models. I also wanted someone else to be able to run the same experiment and get the same numbers.There was another reason for not using a real agent. If I did, I would also be measuring differences in the model, its tool use and the environment around it. API costs would come into the picture too. That would make it harder to tell whether a result came from the workflow I was testing or from the particular agent I happened to use.So I split the problem into two parts.The benchmark pipeline. Only the agent's decisions are simulated.The coding agent benchmark runs seeded, simulated decisions through a harness against a real repository on disk. Every edit, diff and test run is recorded as an event, and the analyzer turns those events into wasted turns, patch churn and final diff size.The repository is real, and the code actually runs. The only thing I simulated is the part that decides what to do next.That matters when it comes to the claims I can make. I can test how a particular harness rule behaves when the agent behaves in a particular way. I can't tell you how Claude Code or Codex behaves, because I didn't run either of them.The tasks. I made ten small bugs, with each one living in its own temporary repository. They cover things like an off-by-one range, a boundary comparison, an empty-list crash, a wrong formula, a missing strip, lost ordering, a slice error, a wrong variable, swapped branch order and a whitespace split.Each repository has a unittest suite and a known reference fix. I also put a plausible wrong fix in the correct file, along with a few harmless changes in unrelated files. I use those as decoys. The tests run in a subprocess, so the agent is dealing with an actual repository rather than a toy list of predefined answers.The agent. The basic policy is pretty ordinary: run the tests, look at the failure, make a change, run the tests again and stop when things pass. I then give different profiles some fixed chances of doing something else along the way.ProfileWrong editRepeats a failing commandRevertsQuits earlyUnneeded edit after a passcareful0.050.050.020.030.05normal0.150.150.080.080.15sloppy0.300.300.200.180.30loop_prone0.100.550.150.050.40These numbers are hypothetical. I chose four behavior profiles, starting with a careful agent and ending with one that is more likely to get stuck in loops. I didn't get those profiles by measuring real agents, so I don't treat the absolute numbers as predictions of what Claude Code, Codex or another agent would actually do. The sweeps later in the article are there partly to check how much the results change when those assumptions change.How the agent reacts to the harness is something I can change directly. compliance controls how often the agent follows a harness instruction instead of ignoring it. plan_benefit controls how much a forced planning step reduces the chance of making the wrong edit.Paired runs. I keep the agent's random stream tied to the profile, task and seed. That means when I compare two harnesses, they start with the same agent behavior. I run them as pairs rather than comparing two unrelated random runs.Success comes from a check the agent doesn't see. Once a run finishes, I run the tests against the final state of the repository. If they pass, the run is successful.What counts as wasted work in an agentic coding workflowI didn't want to count every failed test as wasted work. A failed test can be useful. If I just changed something and the test tells me what went wrong, that result is part of the debugging process.So I count an action as wasted when it doesn't move the run any closer to the final state. In my setup, that includes things like:repeating something the agent has already seen: running the same command again without changing the repository, running a test again when none of the relevant files have changed, or reading the same contents againchanging a part of the repository that has nothing to do with the testsmaking an edit that changes nothing, or making a change that gets undone laterA revert needs a little more care. I don't count the revert itself as waste if it was needed to recover from a bad change. The bad change is what I count.For example: the agent edits the wrong file, the test fails, it checks the diff, reverts the change, edits the right file and the test passes. The wrong edit is wasted work. The revert isn't, because it was needed to get back to a useful state.I also keep harness work separate. If the harness adds a planning step, review step, gate or cleanup step, I count that as a harness turn rather than calling it agent waste. I want to see the cost of adding that structure.Repetition itself isn't unusual. A study of failures in multi-agent LLM systems lists step repetition as one of 14 failure modes [5]. That study isn't about coding agents specifically, but the basic problem is similar.The recorder only stores what happened. It doesn't decide what was wasteful at the time. The analyzer works that out afterwards, which means I can change the definition later without running all the experiments again.The core of it, trimmed:Here are the rules the analyzer uses for tests and edits:By "relevant," I mean a file that the test suite actually imports. The harness can figure that out from the tests. It doesn't need to know what the correct fix is.Basic, structured and adaptive: three agent harnessesI used three different harnesses for the comparison. In this experiment, the harness is simply the layer sitting between the agent and the repository.The three harnesses compared in the benchmark.The basic harness lets the agent work until it says it is done. The structured harness forces plan, implement, test and review. The adaptive harness watches the trajectory, steps in with five rules, and refuses to let the agent finish while tests fail.Anthropic has a useful distinction here. A workflow has the steps worked out in advance. An agent gets to decide what to do next. They also recommend starting simple instead of adding a lot of machinery upfront [3].That is basically how I think about the two harnesses. The structured one tells the agent what to do. The adaptive one doesn't. It waits for certain things to happen and then reacts.There are similar ideas in tools people already use. Claude Code has hooks that can run a command at a particular point in a session, including things like formatting a file after an edit or stopping a command before it runs [4]. SWE-agent also found that the setup around the model matters: how it edits files, moves around the repository and runs tests can affect the result [2].I'm not trying to reproduce either system here. My adaptive harness is much smaller. I just wanted a few fixed rules that could react to what the simulated agent was doing.I locked those rules before starting the runs:RuleFires whenDirectiveR1the same command failed twice with no state changediagnose before editing againR2the same file was modified twice, or two reverts happenedinspect the accumulated diffR3three turns passed with no state changechange strategyR4tests pass after a messy runverify by reviewing the diffR5an edit touched a file the tests do not importjustify it or undo itGatethe agent tries to finish while tests failrefuseTwo of the three harnesses refuse to let the agent finish while tests fail. Keep that in mind. It matters in the next section.Result 1: the cheapest agent harness was misleadingHere is the main comparison. Twenty seeds, ten tasks, 200 runs per cell, compliance 0.5.ProfileHarnessSuccessTurnsWasted turnsHarness turnscarefulbasic92.5%5.20 ± 0.190.29 ± 0.110.00carefulstructured100.0%7.66 ± 0.200.44 ± 0.142.12carefuladaptive100.0%5.76 ± 0.260.28 ± 0.100.42normalbasic77.5%5.25 ± 0.270.63 ± 0.160.00normalstructured100.0%8.82 ± 0.331.31 ± 0.252.35normaladaptive100.0%6.62 ± 0.340.74 ± 0.170.86sloppybasic51.5%5.62 ± 0.411.44 ± 0.270.00sloppystructured100.0%11.91 ± 0.623.15 ± 0.373.10sloppyadaptive100.0%9.95 ± 0.681.97 ± 0.292.83loop_pronebasic89.0%7.45 ± 0.462.46 ± 0.370.00loop_pronestructured100.0%11.59 ± 0.563.85 ± 0.432.48loop_proneadaptive100.0%9.04 ± 0.562.29 ± 0.311.97Table 1. All runs, 200 per cell. Mean ± 95% interval. Created by the author from the benchmark output.At first glance, the basic harness looks like the winner. It has the fewest turns for all four profiles, and in two profiles it also has the lowest waste.But look at the success numbers too.The sloppy agent only solves 51.5% of its tasks with the basic harness. It can simply stop while the tests are still failing. That gives you a short run with very little wasted work, but only because the agent gave up.So looking at turn counts on their own can make a failing harness look better than it really is.For the cost comparison, I only use seeds where both harnesses actually solved the task. Then I compare those runs directly, one against the other.ProfileHarnessMatched pairsExtra turns vs basicExtra wasted turns vs basicHarness turnscarefulstructured185+2.16 ± 0.08+0.12 ± 0.072.04carefuladaptive185+0.23 ± 0.12-0.04 ± 0.050.31normalstructured155+2.52 ± 0.16+0.46 ± 0.152.06normaladaptive155+0.37 ± 0.17-0.01 ± 0.100.49sloppystructured103+3.13 ± 0.29+0.86 ± 0.272.26sloppyadaptive103+1.05 ± 0.51-0.41 ± 0.291.70loop_pronestructured178+3.46 ± 0.27+1.14 ± 0.252.32loop_proneadaptive178+1.10 ± 0.51-0.30 ± 0.321.84Table 2. Seeds where basic and the harness both solved the task, compared run by run. Compliance 0.5. Created by the author from the benchmark output.The picture changes once I line up the successful runs.Structured adds between two and three and a half turns for every profile, including the careful agent. It doesn't reduce waste in any of them either. The waste actually goes up.There may be a simple reason for part of that. When the structured harness sends the agent back from "finish" to do a review, the simulator gives its random deviations another chance to happen. I haven't isolated that effect or tried to correct for it, so I wouldn't put too much weight on the structured waste numbers.Adaptive is much closer to basic for the careful and normal profiles. With the sloppy and loop-prone profiles, it cuts the waste a little. The loop-prone result is still close enough to zero that I wouldn't call it a clear win.The success numbers tell an even simpler story.Both structured and adaptive stop the agent from finishing while the tests are failing. So I tested that rule by itself, without the other five adaptive rules.It produced the same success rate as the full adaptive harness. On seeds where both runs succeeded, it also matched basic exactly: 0.00 ± 0.00 extra wasted turns and 0.00 ± 0.00 extra turns across all four profiles.That result isn't surprising once you look at how I built the simulator. Early quitting is one of the behaviors I put into it. The gate prevents that behavior from ending the run. In this experiment, that one rule accounts for the entire improvement in success.Result 2: only one rule actually cut wasted workSo if the gate explains the improvement in success, what about the five other rules? Do they actually reduce the wasted work?That turns out to depend a lot on whether the agent follows them.The chart below shows the change in wasted turns compared with basic, using matched seeds, as compliance increases.Waste falls as compliance rises. For loop-prone agents the effect only becomes clear at 0.75.On matched seeds, the adaptive harness removes more wasted turns as the agent follows its directives more often. At compliance 1.0 it removes 0.70 turns for sloppy agents and 0.83 for loop-prone agents. An asterisk marks an interval that includes zero.At compliance = 0, the rules still fire, but the agent ignores them. Nothing changes, which makes this a useful control.Once compliance goes above zero, the sloppy profile starts wasting fewer turns at every level I tested. The loop-prone profile is harder to move. I don't see a clear reduction there until compliance reaches 0.75, where the interval finally moves away from zero.The size of the saving is the bigger issue. At compliance = 1.0, adaptive saves 0.70 ± 0.35 wasted turns for the sloppy agent. But it uses 2.99 harness turns to do it, and the complete run is still 1.77 ± 0.68 turns longer than basic.So in this setup, I'm spending roughly three extra turns to save less than one wasted turn.I then took the five rules apart one at a time to see which one was responsible for the saving.Removing R5 is the only change that makes the waste reduction collapse.Each bar shows the change in wasted turns against the basic harness when one rule is removed. The completion gate alone changes nothing on matched seeds. Dropping R5 shrinks the reduction from 0.70 to 0.30 turns for sloppy agents and from 0.83 to 0.26 for loop-prone agents.R5 is doing most of the work here.I found that when I removed it at compliance = 1.0, the reduction in wasted work dropped by more than half for the sloppy profile. For the loop-prone profile, it dropped by roughly two thirds. The other two profiles were different. With R5 gone, the change was slightly positive: +0.11 ± 0.11 for normal and +0.03 ± 0.04 for careful.I also tried taking out R1, R2, R3 and R4. The waste numbers barely moved. Removing R1 or R2 did reduce the extra turn count a little, so those rules seem to be adding overhead without doing much for waste in this particular setup.This is what R5 actually does in the harness:R5 fires when the agent changes a file that none of the tests import. The simulator creates exactly this situation with its decoy edits.That is also the part where I had to question my own result. I created the decoy edit, and then I created the rule that catches it. So is adaptive actually better, or did I just give it a problem that I knew how to detect?I think a skeptical reader is right to ask that. I asked it too.The next section is my attempt to check it. One caveat up front: I designed the rule ablations after seeing the first results, so they are post hoc rather than part of the original plan.Result 3: what happens when the rules don't cover the behaviorBefore I ran the held-out tests, I wrote down what I expected to happen. This is the basic idea behind preregistration: decide on the questions, predictions and analysis before looking at the results, rather than coming up with an explanation after the fact [6]. I borrowed that approach here. The predictions are in a file in the repository, and I froze it before running the held-out suite.For the held-out runs, I added four new agent profiles. None of them edits a file outside the module being tested, and none of them quits while the tests are failing. That means R5 has nothing to catch, and the success gate has nothing to do.I also added a unit test to make sure R5 really cannot fire on these profiles.held_oscillate. The agent makes a wrong edit, reverts it, and then has a 0.6 chance of making the same wrong edit again.held_rerun. After the tests pass, the agent runs the same passing tests again, up to three times.held_core_thrash. After getting a pass, the agent makes a pointless cosmetic change inside app/core.py. R5 doesn't see this, and neither does my file-level definition of waste.held_mixed. A combination of the three behaviors above.There is a catch here. Two of these behaviors are related to rules I had already written. R2 was meant to catch oscillation, and repeated passing tests can trigger R3. So this isn't a completely untouched test of the rules. It is more of a stress test.I couldn't come up with a behavior that all of the rules would miss, because the rules themselves are fairly general.My prediction was simple: adaptive would show no demonstrated reduction in waste on any of these four profiles.I used 50 seeds, 10 tasks and 500 runs per cell, with only matched seeds included in the comparison.ProfileCompliancePairsAdaptive: wasted turnsAdaptive: total turnsAdaptive: extra diff linesStructured: wasted turnsStructured: total turnsheld_oscillate0.5489-0.91 ± 0.27-0.32 ± 0.31+0.00 ± 0.00+0.00 ± 0.00+2.00 ± 0.00held_oscillate1.0488-1.67 ± 0.36-0.42 ± 0.37+0.00 ± 0.00+0.00 ± 0.00+2.00 ± 0.00held_rerun0.5500+0.00 ± 0.04+0.19 ± 0.06+0.00 ± 0.00+0.71 ± 0.08+2.71 ± 0.08held_rerun1.0500+0.03 ± 0.04+0.48 ± 0.10+0.00 ± 0.00+0.71 ± 0.08+2.71 ± 0.08held_core_thrash0.5500+0.48 ± 0.05+0.95 ± 0.11-0.45 ± 0.04+0.29 ± 0.04+2.77 ± 0.07held_core_thrash1.0500+0.66 ± 0.05+1.43 ± 0.12-0.60 ± 0.04+0.60 ± 0.04+3.07 ± 0.06held_mixed0.5499+0.05 ± 0.12+0.65 ± 0.20-0.21 ± 0.04+0.66 ± 0.07+3.05 ± 0.11held_mixed1.0499-0.09 ± 0.17+1.18 ± 0.25-0.29 ± 0.04+0.82 ± 0.08+3.21 ± 0.11Table 3. Held-out profiles, 50 seeds, matched seeds only, difference vs basic. Created by the author from the benchmark output.The prediction was right for three of the four profiles and wrong for one.held_rerun and held_mixed did what I expected. Adaptive didn't reduce waste, and it added turns.held_core_thrash was more interesting. Adaptive increased the amount of work my analyzer called waste, but at the same time it removed stray lines from the final patch. I'll come back to that distinction below.The miss was held_oscillate. I expected the agent to keep going around the same loop because R2 only looks at the diff. Instead, adaptive reduced waste by 0.91 ± 0.27 turns at compliance = 0.5 and by 1.67 ± 0.36 at compliance = 1.0. It didn't add any total turns either.So I looked at what was happening.There was an assumption in my simulator that I hadn't noticed. When the harness reverted the stray edit, the simulated agent also forgot that it had wanted to make that edit again. That makes the harness look better than it should.I changed the simulator so that the agent still wants to re-apply an edit after the harness reverts it, then ran the case again.Agent after a harness revertWasted turns, adaptive minus basic (0.5 / 1.0)Total turns, adaptive minus basic (0.5 / 1.0)Forgets the edit (pre-registered)-0.91 ± 0.27 / -1.67 ± 0.36-0.32 ± 0.31 / -0.42 ± 0.37Still wants it-0.62 ± 0.25 / -1.21 ± 0.32+0.15 ± 0.33 / +0.59 ± 0.43The waste reduction gets smaller by about a third, but it doesn't disappear. The advantage in total turns does disappear, though.I didn't break down the remaining reduction to see exactly where it comes from, so I’m treating this rerun as exploratory. The original preregistered results stay unchanged.Final patch size vs total work: the same run through three harnessesThe core-thrash result made me look at another part of the run. Counting wasted turns tells me what happened along the way, but it doesn't always tell me what was left behind in the final patch.So let's go back to the run from the start of the article: the loop-prone agent, the off-by-one task, seed 6, with compliance = 0.5.Same agent, same task, same seed.In this run, the basic harness returns an eight-line patch with eight lines of churn in 11 turns. The structured and adaptive harnesses return the two-line fix, but with 14 lines of total churn, because they apply stray edits and then revert them.The basic harness leaves an eight-line patch, including three edits that have nothing to do with the task. Structured and adaptive leave the two-line fix.But they do more work to get there. Both go through the stray edits and then undo them, so the total churn is 14 lines.That creates a funny result. If I only look at turns, basic wins. If I look at the final diff, basic loses. Neither number is wrong. They are measuring different parts of the run.The same thing shows up across the matched seeds. For sloppy agents, adaptive leaves a final patch that is 0.97 ± 0.35 lines smaller on average. For loop-prone agents, the difference is 0.71 ± 0.29 lines. But total churn goes up by 0.66 ± 0.54 and 1.00 ± 0.45 lines respectively.The held_core_thrash case makes the problem even clearer. Adaptive can remove a pointless line from the final patch and still get a higher waste score. The reason is how I defined waste: once the harness cleans up that edit, it becomes an edit that was "reverted later", and that gets counted as waste.So one metric says the run got worse while another says the final patch got cleaner.I don't think that is a bug in either metric. It is a good reason not to squeeze everything into one efficiency score.Design decisions I would defend, and the ones I would notThere are a few choices in the experiment that I think are worth making explicit. Change any of them and some of the numbers change too.Paired seeds. Every harness starts with the same agent behavior for a given seed. That removes a lot of random noise from the comparison. Without paired runs, the intervals would be wider and I would be much less confident about the differences.Harness turns are separate from agent waste. A review step is work, but it isn't something the agent decided to do. Counting it as agent waste would make structured look worse for the wrong reason. Leaving it out completely would make those turns look free. So I report the two separately.The success check is hidden. The agent doesn't get to decide whether it solved the task. Once the run ends, I run the tests on the final repository state. If it quit while the tests were failing, that's a failure.The recorder stores facts, not judgments. It records what happened and leaves the waste labels to the analyzer. That means I can change the waste definition later without running the whole experiment again.Test results are cached. If the repository is unchanged, the test result should be the same, so I only run each distinct repository state once. I have a unit test that compares a cached result with a fresh subprocess run, and --no-cache disables the cache when I want every test run to execute for real.There are also four choices I wouldn't defend nearly as strongly.One probability for compliance. Real agents don't have a single setting that says how often they follow a directive. I use one number because it makes the sweep easy to understand. The fact that the results move as that number changes is useful, but the model is still crude.File-level relevance. I call a file relevant when the tests import it. That works for the decoy edits I wanted to catch, but it has an obvious hole. A pointless edit inside the correct file isn't caught by either R5 or my waste definition.The agent forgetting. In the first version of the simulator, once the harness reverted an edit, the agent forgot that it had wanted to make that edit. That made the oscillation result look better than it should have. The rerun with a persistent intention reduced the effect.The 40-turn cap. This one matters for the oscillation profile. At compliance = 0.5, every unsuccessful run hit the cap: 11 of 500 for basic, 15 of 500 for structured and none for adaptive. If I increased the cap, those success rates could change. So I wouldn't treat the exact success numbers there as stable beyond this experiment.Limits of this coding agent benchmarkThere are some things this experiment simply cannot tell me. I don't want the results to sound broader than they are.The agent is simulated. I didn't run Claude Code, Codex or another real coding agent here. A real agent could behave in ways none of my profiles capture.The profile and compliance numbers are mine. I varied them and ran sweeps over compliance, but the values themselves aren't measurements of real agents.The tasks are tiny. I used ten bugs, each affecting a single function. Real repositories bring much messier problems: changes across several files, flaky tests, slow builds and other things this setup doesn't model.Waste is defined at the file level. If the agent makes a pointless change inside the correct file, my analyzer doesn't know that it was pointless. The held_core_thrash result shows this pretty clearly.I may still have simulator assumptions I haven't noticed. The oscillation result found one for me. I hadn't realised that the simulated agent forgot about a change after the harness reverted it.The ablations came after the first results. The held-out suite also overlaps with behaviors that R2 and R3 were designed to catch. I've already called this out above, but it matters enough to repeat.The intervals have limits too. I use normal-approximation intervals on paired means across a lot of cells. With this many comparisons, some results will look interesting just by chance, and success rates are bounded proportions. I therefore put more weight on patterns that show up across profiles and compliance levels than on any single cell.What to measure in your own agent workflowIf you're running coding agents on real work, these are the things I'd take from my experiment. They all held in my simulation, but that doesn't mean they'll hold in yours. I'd check them rather than assuming.Keep final patch size and total churn separate. A small final diff can hide a lot of edits that happened along the way.Compare cost only where both setups succeed. Otherwise, the setup that gives up early can look cheaper simply because it stops doing work.Don't let the agent declare success while the check is failing. This was the cheapest rule I tested, and it made the biggest difference to success.Test rules on problems you didn't build them around. If you write a rule to catch a failure and then only test that failure, the result isn't very convincing.Write down what you expect before running the experiment. One of my predictions was wrong. Having it written down meant I couldn't quietly change the story after seeing the result.From simulation to a real agent: how I would test this nextEverything so far comes from the simulator. I haven't run this setup on a real coding agent yet.The first thing I'd need is a recorder that captures a few details for every action: what the agent did, what it touched, the command it ran, the exit code, an output hash, the repository state before and after, and the lines added or removed.Claude Code hooks can run your own commands at different points in a session, including before a command runs or after a file edit [4]. That should be enough to turn each event into one JSON line. I could then map those events onto the same structure used by this experiment and run the existing analyzer without changing it.There are two parts I'd be careful about.First, "relevant file" needs a better definition for a real repository. I used the files imported by the tests. On a real project, I'd probably need an import graph or the files changed by a known reference patch.Second, a real agent won't give me the same trajectory every time. I'd need many runs of the same task under each harness before comparing them. One run can show that something happened. It can't tell me how often it happens.Run the benchmark yourselfYou only need Python 3.10 or newer to run the benchmark. There are no extra packages to install because it uses only the standard library.I also seed every random choice, so if you run the same benchmark, you should get the same numbers I did.The main run takes about a minute on my laptop. The full compliance sweep (--sweep) takes about four minutes. The last command shows the turn-by-turn trajectories, including the actual diffs. The pre-registration file, HELDOUT.md, is in the repository alongside the code.Repository: https://github.com/Emmimal/agent-efficiency-benchmark/Conclusion: the pass rate leaves out the workA coding-agent benchmark that only records whether the final patch passes misses a lot of what happened before that patch. In the run that opens this article, the tests passed, but the agent still made six wasted turns and left a final patch four times larger than the reference fix.Once I started recording the trajectory, the cost of the harness showed up too. In this simulation, one rule reduced wasted work but added turns. Another harness left a cleaner final patch while making the waste metric worse. You don't see either of those things in a pass rate.I wouldn't take these results as a verdict on a real agent or a real coding workflow. I see them as a reason to log more than just whether the final patch passed.DisclosureI have no affiliation with, sponsorship from or financial interest in Anthropic, Princeton University or any other organization or tool named in this article. No real coding agent and no language model was run for this study. Every agent decision is simulated, and I say so wherever results are reported. I ran every experiment and checked every number in the text against my own output files. All diagrams in this article, including the featured image, were created by the author. The featured image was generated with ChatGPT (DALL·E).Resources[1] Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? International Conference on Learning Representations (ICLR 2024). arXiv:2310.06770. https://arxiv.org/abs/2310.06770[2] Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv:2405.15793. https://arxiv.org/abs/2405.15793[3] Anthropic. (2024). Building effective agents. Engineering at Anthropic. https://www.anthropic.com/engineering/building-effective-agents[4] Anthropic. Hooks reference. Claude Code documentation. https://code.claude.com/docs/en/hooks (accessed October 2026).[5] Cemri, M., Pan, M. Z., Yang, S., et al. (2025). Why Do Multi-Agent LLM Systems Fail? Advances in Neural Information Processing Systems (NeurIPS 2025), Datasets and Benchmarks Track. arXiv:2503.13657. https://arxiv.org/abs/2503.13657[6] Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600-2606. https://doi.org/10.1073/pnas.1708274114[7] Alexander, E. P. (2026). Agent efficiency benchmark: code, pre-registration and results. GitHub. https://github.com/Emmimal/agent-efficiency-benchmark/