When an agent's policy is cut into separate skills that it loads one at a time, a task can need a clause that lives in a skill the agent never opens. In a paper posted on 11 November 2026, a Gemma 4 12B agent that read the airline policy of τ²-bench from on-demand skills scored 56.0 pass^1, against 67.5 for the same untrained model in a reference harness with the whole policy in its system prompt: 11.5 points lower. Here pass^1 is expected per-trial success over 50 official airline tasks run 4 times each, scored by the benchmark's check of the final database and conveyed information, with an LLM-simulated customer. In retail, with self-contained policy sections, the split harness scored 72.6 against 73.2. We have not reproduced or run anything; every number below about the paper is the authors'.
What was held fixed and what varied
The paper's main subject is an agent that improves without a stronger teacher: it writes rules into its skills from failed conversations and trains a LoRA adapter with GRPO reinforcement learning on its own conversations.
For the counterexample, the untrained model, the benchmark and the user simulator are fixed; what varies is the harness. Besides moving the policy into skills, H0 rewrites the agent instruction, adds a load_skill tool and a skill catalog with loading rules shared by both domains, and appends a strict-follow notice to each loaded document; the paper does not separate these factors. The reference agent holds the whole policy in its system prompt. The starting system, H0-M0, gets the same policy split verbatim into an entry document plus one skill per kind of database write (airline: book, modify, cancel, compensation), with a catalog rule to load the most specific applicable skill. The authors locate the defect in that harness, not in the model.
Source: Yujia Zhang, Xinqi Ren, Shixiong Zhou, Jia Meng, "Knowledge in the Harness, Reliability in the Weights: Teacher-Free Co-Evolution of an LLM Agent", arXiv:2611.15268v1, abstract, sections 3.2, 3.3 and 5.5, Appendix D, arxiv.org/abs/2611.15268v1.
How success is counted
A conversation scores 1 only if the final database matches the reference state and the required information was conveyed. The user simulator is DeepSeek-V4.1-Flash. Airline's required information is checked by substring match; GPT-4.1 judges retail's natural-language assertions. The official sets are 114 retail and 50 airline tasks, each run 4 times.
The authors call this a custom setting whose absolute scores are not directly comparable to the public leaderboard.
Source: the same work, sections 3.1, 3.2 and 3.3, Appendix A and Limitations, arxiv.org/abs/2611.15268v1.
Where the missing obligation lived
The airline policy defines the free baggage allowance, the $50 fee per extra bag and the insurance coverage in its booking section, yet modifications and cancellations need them. According to the authors, the verbatim split, together with the rule to load the most specific skill, put those clauses out of reach for a modify or cancel task that never loaded the booking skill.
The 11.5-point airline gap (Table 2) comes with a 95% paired bootstrap interval over tasks of 4.5 to 19.0 points below the reference. Retail, whose sections the authors describe as self-contained, showed no such gap.
The authors illustrate the mechanism with ten airline baggage tasks (40 conversations per system, Table 13). H0-M0 loaded the booking skill in 30% of its 40 conversations. It succeeded in 6 of 12 conversations that loaded it and in 0 of 28 that did not. The reference agent succeeded in 18 of 40.
Source: the same work, sections 5.1 and 5.5, Table 2 and Table 13 (Appendix E), arxiv.org/abs/2611.15268v1.
How the gap was closed
The agent's own editing loop found the defect: its logs record which skills were loaded before each write, so it labelled the failures "not loaded". It added a routing rule to load both the modify and book skills when baggage changes, copied the allowance formula into the modify skill, and added the insurance condition to the cancel skill.
Booking-skill loads rose only from 30% to 40%, but conversations that skipped the skill began to succeed (6 of 24), and those that loaded it went from 6 of 12 to 12 of 16. In our reading, on these ten tasks much of the gain came from the clause being present where it is used.
Across all 50 airline tasks, the system with round-1 rules and untrained weights (H1-M0) reached 67.0 pass^1, against the reference agent's 67.5.
Source: the same work, section 5.5, Tables 4 and 5 (Appendix B) and Table 13 (Appendix E), arxiv.org/abs/2611.15268v1.
The training results, bounded
After two rounds of rules and training, pass^1 moved from 72.6 to 85.1 in retail (12.5 points) and from 56.0 to 75.5 in airline (19.5 points); the authors report all intervals above zero. In their ablation (Table 3, independent runs of 4 trials per task), on tasks the starting system solved in none of 4 trials (8 retail, 15 airline; the authors note retail's bin is small), final rules with base weights reached 44% and 27% mean success, against 25% and 5% for trained weights with the original skills. On tasks it solved in all 4 (55 retail, 19 airline), the weights alone held 95% and 99%, the rules alone 87% and 93%. The authors report that 14 of the 15 never-solved airline tasks depend on the skill split.
The authors state the limits: one base model, two domains of one benchmark, two rounds; most single updates are not individually significant.
Source: the same work, sections 5.2 and 5.4, Tables 2 and 3, and Limitations, arxiv.org/abs/2611.15268v1.
What we are doing with it
On 18 November our research desk read this paper and proposed a bounded trial: about 20 synthetic authorised work items with cross-skill requirements (for example modify versus create, access and retention policy, cancellation and the original contract), a fixed solver, evidence and budget. It would compare the current flat policy with a versioned dependency map, measuring coverage of required clauses, an independently checked end state, unauthorised acts, refusals and UNKNOWN, latency and tokens. A missing critical inherited clause, privilege escalation or unrecoverable regression would be a fail.
The recommendation is a trial for dependency and closure guards, and a hold on any GPU retraining. Status: proposed. No owner has decided. Nothing was run, trained or reproduced.
One point is our interpretation, not the paper's: a retrieved low-trust skill must not gain authority over verified system constraints.
Source: the same work, arxiv.org/abs/2611.15268v1.