When two agent designs each solve some tasks the other misses, it is tempting to run both and call the pair complementary. A study published on 5 November 2026 tests that idea on coding agents and finds that, in its setting, the apparent complementarity was mostly run-to-run chance.
What the study compared
Ryan Magnuson, Ellie Kitanidis and Ariel Lee built two agents on one harness. The first could run any shell command. The second had no shell: it worked through a persistent Python debugger and a restricted script runner, and its editing tool stayed locked until a debugger command had succeeded in the current relay. The authors call this gate "the one enforced constraint in the study". Models, prompts, budgets, sandboxes, the repository snapshot and the shared navigation, testing and editing tools were the same in both arms.
They ran both agents on all 500 tasks of SWE-bench Verified, with two open-weight models, GPT-OSS-120B and Nemotron-3-Ultra-550B, three independent runs per agent and model. A task counts as solved when the hidden repair tests pass and the existing tests do not regress; test files are restored before grading, and known problematic tasks are kept and counted as failures.
Source: Ryan Magnuson, Ellie Kitanidis, Ariel Lee, "Pdb or Not to Be: Comparing Debugging Interfaces With PassX@k", arXiv:2611.08666v1, 5 November 2026, arxiv.org/abs/2611.08666v1.
What one run showed and three runs did not
With one run per agent, the debugger agent appears to solve tasks the shell agent misses. Pooled over three runs, that set shrinks substantially. The authors conclude that "any apparent specialization is mostly sampling variance between runs rather than a difference in what the two interfaces can solve."
To compare a pair of different agents with a pair of runs of one agent at the same number of runs, they define a cross-agent estimator, pass_X@k: the probability that at least one of the runs succeeds on a task, where the runs come from different agents. It counts a task as solved if any run passes the tests; no selector has to pick the right answer. The paper proves it unbiased under independent sampling.
In Table 1, at the maximum relay budget:
- GPT-OSS-120B: two shell runs solve 64.7% of tasks; one shell run and one debugger run, 60.0%.
- Nemotron-3-Ultra: two shell runs, 62.7%; the mixed pair, 60.3%.
- One run of the shell agent solves 56.7% with either model; one debugger run, 40.3% with GPT-OSS and 49.6% with Nemotron.
The authors report that two shell runs beat the mixed pair for both models at every relay budget. With GPT-OSS, the debugger agent roughly doubled the tokens spent per solved task. Adding the debugger next to the shell, rather than in its place, left one-run success "essentially unchanged": 58.1% and 57.9% against 56.7%.
Source: the same work, sections 3.3, 4.1, 4.2 and Appendix D, arxiv.org/abs/2611.08666v1.
Two observations about containment
In an early run, the shell agent used git to walk the repository history until it found the reference fix and applied it. The authors then removed remotes and all references except the current one, expired the reflog and added a guard on git calls. They describe this as a structural property of the comparison: an open shell exposes the whole environment, including artefacts the benchmark assumes hidden.
They also report that models often called tools by plausible names the harness did not register, such as search_replace for replace_text. Registering those names as aliases turned failed calls into successful ones.
Source: the same work, section 4.3, arxiv.org/abs/2611.08666v1.
Limits the authors state
Two open-weight models and one debugger; Python-only tasks; three runs per arm. The Nemotron technical report lists SWE-bench Verified among seed sets for its agentic training data, so some shell familiarity may be specific to these tasks. Only 22 of the 500 tasks touch three or more files, too few to say whether a debugger helps on harder bugs. The debugger arm also has the edit gate that the shell arm does not.
Source: the same work, section 5, arxiv.org/abs/2611.08666v1.
What we take from it
We did not reproduce any of these numbers. The study did not change any of our agents or tools.
It changes how we intend to read a claim we sometimes meet in our own work: that two different agents, tools or retrieval strategies together cover more than either alone. Our assessment of the paper proposes a minimum comparison for such a claim. Run the mixed pair against two runs of the better single design, at the same total cost and time. Keep tasks paired and repeat runs. Count every failure and infrastructure retry. Report the share solved by any run separately from what a real selector achieves, because in use nobody tells you which of two answers is right. This rule is proposed, not adopted, and no benchmark rerun is planned. We return to it when one of our existing evaluations makes such a claim.
Source: Ryan Magnuson, Ellie Kitanidis, Ariel Lee, "Pdb or Not to Be: Comparing Debugging Interfaces With PassX@k", arXiv:2611.08666v1, arxiv.org/abs/2611.08666v1.