ARCANADA
All Posts
Blog November 13, 2026

Three agents in debate proposed fewer wrong links than one, and missed more right ones

Which files implement this requirement? Agents are now asked that question directly, with the whole repository in context. A new study measures how one agent answers it, and what changes when further agents criticise and revise the answer or three agents argue about it. We have not reproduced the results. Below is what was measured and what the word "cost" means in the paper, because it does not mean money.

One agent, the whole repository, no retrieval step

Sarthak Banerjee, Sallam Abualhaija, Domenico Bianculli and Lionel Briand start from a known weakness of the usual pipeline: files are first retrieved by similarity and only then classified, so a relevant file that retrieval misses can never be recovered. Their single-agent design skips retrieval and gives the model one requirement and the complete code base.

They test on three public Java projects with ground-truth links: a tourism system with 58 requirements, 116 code files and 308 links; a healthcare system with 131 requirements, 226 files and 286 links; and a drone-control system with 57 requirements, 184 files and 393 links. One model version is used throughout, with temperature set to zero. The comparison point is a published retrieval-based method run with the same model.

A link counts as correct when it is in the project's ground truth. On average over the three projects, the single agent recovered 80.9% of the true links, 38.6 percentage points more than the retrieval baseline. Its precision fell by 25.8 points, to 17.8%: of the links it proposed, fewer than one in five were in the ground truth.

Source: Sarthak Banerjee, Sallam Abualhaija, Domenico Bianculli, Lionel Briand, "Rethinking Requirement-to-Code Traceability Through Single-Agent and Multi-Agent Approaches", arXiv:2611.09162v1.

A second opinion removes wrong links and some right ones

The first multi-agent design uses three roles: a tracer proposes candidate links, a critic flags weak evidence, and an author revises those initial proposals. Precision rose from 17.8% to 26.9%. Recall fell from 80.9% to 66.5%. The design can improve precision, but it cannot recover genuine links that were never proposed in the initial set. The authors call this design less attractive in practice than the single agent: more computation for a small gain.

The second design is a debate. One agent argues which files trace to the requirement, another argues which do not, each sees the other's output once, and a judge produces the final list. All three receive the full code base. With the more detailed of two prompts, the debate reached 24.5% precision and 71.0% recall. Against the single agent with the same prompt, that is 6.7 points more precision and 9.9 points less recall. The abstract reports a combined score that weights recall above precision of 49.5% for this design, 7.5 points above the retrieval baseline.

The debate also shortened the list a person would have to check: from about 17% of all possible requirement-file pairs to 12% on the tourism project, from 7% to 5% on the healthcare project and from 14% to 7% on the drone project. The price was missed true links: the authors report that about 17% of the ground-truth links on the first project and 50% on the third were not recovered. That share is taken of the true links, not of all possible pairs.

Source: the same paper, section IV-H and the abstract, arXiv:2611.09162v1.

What "cost" means in this paper

The cost column is the share of all possible requirement-file pairs that an analyst would have to inspect if they reviewed only what the method returned. It is a measure of human review load, and the authors note that it assumes every link takes equal effort to inspect.

It is not the price of running the agents. We searched the text for token counts and prices and found none; the only remark on compute is that the three-role design costs more than the single agent. So the paper does not show whether three agents beat one agent given the same budget. The debate runs the full code base through three agents and two rounds; a single agent given that budget was not tested.

The authors name further limits. All three systems are written in Java, and the authors call for studies on other languages and larger systems. One commercial model was used, and its outputs may vary between runs. The qualitative part of the paper suggests where each design fits: retrieval when a requirement repeats the vocabulary of the code, a single agent when code is well documented and consistently named, debate when the evidence is indirect, conflicting or spread across files.

What this means for our work. Our agents are asked to connect a requirement, the change that implements it and the evidence that it works. This study says what to expect from an agent's list of such links: high coverage comes with many wrong entries, and adding further agents trades coverage for a shorter list. Such a list is a set of candidates to check, not a statement of coverage. A trial on one of our own repositories, with a deterministic search, one agent and a debate compared on the same budget, has been proposed and is on hold: nobody has accepted it and nothing has been run.

Source: the same paper, sections IV-D, IV-I and V, arXiv:2611.09162v1.