CWE-bench: Refocusing Agentic Security from Exploitation to Defense
Top AI agents patch only 68% of injected vulnerabilities on first try when not told where the bug is.
Researchers introduce CWE-bench, a 120-task private benchmark testing whether AI agents can audit and repair vulnerabilities across 73 CWE types, eight languages and the OWASP Top 10 2025, without being told what or where the flaw is. The best agents achieve 68% pass@1 and 81% pass@4, and the study finds no single agent dominates across languages or vulnerability categories, with difficulty driven more by vulnerability type than language.
Why it matters
Most agentic security evaluation has focused on offense (can an agent find/exploit a bug), but defense is a harder bar since a defender must catch everything an attacker needs only one hole. This benchmark gives the field a way to measure whether agentic defensive capability is keeping pace with offensive capability, which matters as AI agents get deployed to both attack and patch code.
What is verified
CWE-bench is a private benchmark of 120 held-out repository audit-and-repair tasks spanning 73 CWE types, eight languages, and all OWASP Top 10 2025 categories. [1]
Agents are given a repository and threat model but not told the weakness, its location, or how many vulnerable sites exist, and patches must block the exploit while preserving functionality as checked by a programmatic verifier. [1]
The best agents tested reach 68% pass@1 and 81% pass@4 on the benchmark. [1]
Agents with the same pass@1 differ widely in retry gains and cost, and no single agent is best across all languages or vulnerability categories; difficulty varies more by vulnerability category than by language. [1]
What is not
- Which specific agents/models were tested is not stated in the abstract.
- The verifier methodology details and how disagreements with agentic judges were resolved are only summarized, not fully specified.
- No information on whether the benchmark or code is publicly released for independent use.
- No comparison to prior defensive benchmarks is given to establish how much of an advance this represents.
Sources
- 1arXiv cs.CRarxiv.org · Primary source
Original source published 12 October 2026 at 18:00.
An AI-assisted briefing by Coding Partners. We link the evidence behind the reporting so you can explore the original work and assess the claims. How CP Signal works.