On the Reliability of LLM-Based Vulnerability Patching Benchmarks
A new arXiv paper argues that benchmarks for LLM-based automated vulnerability patching are unreliable: prompting and tool setup can inflate PoC-passing success without producing developer-quality fixes. Using 112 historical bugs across 84 open-source C/C++, Go, and Rust projects with richer developer-aligned tests, the authors find that while PoC pass rates look strong, developer-test pass rates stay low and barely improve with newer models.
Why it matters
This is a methodology critique that matters to anyone citing LLM vulnerability-patching benchmark numbers: it shows that commonly reported success rates may reflect symptom suppression rather than root-cause fixes aligned with developer intent, and that evaluation design choices can swing results significantly.
What is verified
The authors curated 112 historical bugs from 84 open-source C/C++, Go, and Rust projects, each with PoC tests, regression tests, and additional developer tests assessing alignment with original developer design principles. [1]
The paper identifies agent-level, framework-level, and dataset-level pitfalls that can inflate or suppress measured LLM vulnerability-patching success rates. [1]
LLMs achieve high proof-of-concept (PoC) passing rates under ideal conditions, but developer-test passing rates remain low and improve only marginally with newer models. [1]
What is not
- No specific numeric pass rates, model names, or comparative benchmark scores are given in the abstract.
- The full paper's practical guidelines for evaluation are not detailed in the available text.
- It is unclear which commercial or open LLM tools were tested, or whether findings apply to agentic coding tools like Claude Code or Codex used in client work.
- No peer review or independent replication of these findings is mentioned.
Sources
- 1arXiv cs.LGarxiv.org · Primary source
Original source published 7 October 2026 at 16:26.
An AI-assisted briefing by Coding Partners. We link the evidence behind the reporting so you can explore the original work and assess the claims. How CP Signal works.