Two things in an agent system are easy to miss because nothing fails when they go wrong. A weak plan still produces a result. A retried call may eventually return an answer. Two recent papers measure both. We have put both on hold, and below we say what would bring us back.
Writing the plan is a skill of its own
Zheng Jiang, Fang Niu, Bo Peng, Houde Qian, Wenyi Li, Deyao Hong, Mingju Gao, Yizhe Chi, Youjie Zheng, Kaisen Yang, Calvin Xiao, Lifeng Sun and Huan-ang Gao built WorkflowArena: 145 long tasks in 8 professional domains, from software engineering and formal mathematics to legal analysis and 3D production. Each task gives the agent a workspace and asks for one deliverable made of parts that depend on each other. The agent decides by itself whether to hand work to other agents and how. Every handover is recorded.
A task counts as resolved when the submission meets all its acceptance criteria, as judged by a verifier the agent cannot see. By that measure the strongest of the 25 tested configurations resolved 36.95% of tasks, averaged across domains. No configuration was best in every domain.
The part that matters to us is a second experiment. On a subset of 32 hard tasks the authors fixed who does the work and changed only who writes the plan. A source model wrote the workflow without touching the workspace; every worker was a weaker model. Plans from one source raised the score of all four worker models, by 6.41 points on average when the plan was written once in advance and by 6.27 when it could be revised. Plans from the other source, a model that by itself resolves far more tasks than any of the workers, changed the average by -2.53 and +1.34 points and lowered the score of two workers in both modes. These are points on a continuous score, not shares of solved tasks.
The authors list the limits and we repeat them. Each configuration ran once per task, so close differences have no variance estimate. The subset is 32 tasks and two sources. Content and 3D deliverables were graded by a language-model judge whose agreement with people was not measured. And the plan includes the instructions written for each worker, so a better result may come from better organisation or from task knowledge passed along in those instructions.
What this means for us. When one agent splits a task and others carry it out, we tend to credit or blame the executor. This work gives a way to ask the other question: hold the executor, the starting state and the budget fixed, change only the plan, and let an independent check decide. We have not run such a comparison. The item is on hold in our research roadmap until an orchestration comparison is actually assigned, or until a version of the benchmark with repeated runs appears. Nothing was reproduced by us.
Source: Zheng Jiang, Fang Niu, Bo Peng, Houde Qian, Wenyi Li, Deyao Hong, Mingju Gao, Yizhe Chi, Youjie Zheng, Kaisen Yang, Calvin Xiao, Lifeng Sun, Huan-ang Gao. WorkflowArena: Benchmarking Executable Workflow Orchestration in LLM-Based Multi-Agent Systems. arXiv:2611.12856v1, 9 November 2026.
A retry can cause another execution
Xindan Zhang and Fei Wang looked at what happens when a client gives up on a slow model call and sends it again. The client has left. Unless the retry is merged with the execution already running, the engine may keep computing the abandoned first attempt while it starts a second execution.
They overloaded a vLLM engine for a short time on three open models and then let calls arrive below its capacity. With retries on, passing the cancellation through to the engine did not restore service in any run: of the calls that arrived after the overload, 12 to 38% were delivered. Scheduler traces of one model show where the work went. Executions that the engine later aborted took 79 to 85% of the prompt tokens it computed, and the engine's own prompt-token counter missed a third to a half of all the prompt tokens it computed.
Two controls worked differently. A retry budget restored service by dropping retries, and left many calls from the overload undelivered. Merging a retry with the execution already running for the same call delivered every call, some late, unless cancellation had already aborted that execution.
The limits are stated by the authors: one engine version, one type of GPU, synthetic single-turn prompts.
What this means for us. Our agents reach models through several layers, and more than one of them can retry. The paper's distinction is the useful part: a logical call, an attempt to deliver it, and an execution in the engine are three different things, and a cost or a failure belongs to one of them. We have not changed retry or cancellation behaviour anywhere. The proposed first step costs nothing: read the logs we already have and see whether the three can be told apart and which layer retries. The item is on hold until a measured timeout incident or a found case of two layers sharing one retry budget.
Source: Xindan Zhang, Fei Wang. Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine. arXiv:2611.08684v1, 5 November 2026.
What comes next
Both items share one habit we want to keep: before saying that something helped, name what was held fixed and who decided the outcome. For plans, that is the executor and the check. For retries, that is the level at which the cost was counted. We will report here when either item leaves hold, and with what measurement.