SWE-Residual: Evaluating Starting-State Robustness of Coding Agents
Coding agents lose up to 53.5 points of success when they inherit unfinished work instead of starting clean, and rankings flip.
A new arXiv benchmark, SWE-Residual, tests coding agents not from a clean repository state but from a state where a predecessor agent already attempted the task and (wrongly) reported it complete. Across five frontier agents, success dropped 20.6-53.5 percentage points on DeepSWE and 20.6-34.9 points on Terminal-Bench versus clean starts, with agent rankings reversing on DeepSWE. The authors attribute most failures to 'acceptance bias': agents treat inherited work as the definition of the remaining task rather than re-checking it against the original request.
Why it matters
Most coding-agent benchmarks measure performance from a clean slate, but real-world agent handoffs and multi-step/multi-agent workflows routinely involve continuing someone else's incomplete work. This paper shows that is a distinct, under-measured capability gap: agents are much worse at auditing and correcting prior work than at doing it from scratch, and current leaderboard rankings may not reflect this.
What is verified
SWE-Residual is a new benchmark with 198 Residual States drawn from 111 tasks in DeepSWE and Terminal-Bench, where the starting state already contains an earlier agent's incomplete work rather than a clean state. [1]
Across five frontier agents tested, success rates dropped by 20.6 to 53.5 percentage points on DeepSWE and 20.6 to 34.9 points on Terminal-Bench compared to a Clean Start on the same tasks, and agent rankings on DeepSWE reversed. [1]
The paper identifies an 'acceptance bias' where, in 58.9-88.2% of failed runs on DeepSWE, agents submitted with exactly the same hidden tests failing as the predecessor's state, despite passing most of those tests when starting clean. [1]
What is not
- Which five frontier agents were evaluated is not specified in the abstract.
- The paper is under review and not yet peer-reviewed; no independent replication is available.
- No information on whether mitigation strategies (e.g., prompting agents to re-verify inherited work) were tested or how effective they might be.
- Full methodology and task selection criteria beyond the abstract are not available from this evidence.
Sources
- 1arXiv cs.AIarxiv.org · Primary source
Original source published 19 October 2026 at 09:00.
An AI-assisted briefing by Coding Partners. We link the evidence behind the reporting so you can explore the original work and assess the claims. How CP Signal works.