ProtoArena: Can AI Agents Discover and Exploit Real-World Cross-Component Protocol Vulnerabilities?
An AI agent benchmark shows LLMs can already exploit a chunk of real protocol flaws with guidance, but only 12.5% blind.
Researchers introduce ProtoArena, a benchmark of 80 controlled, authorized environments testing whether LLM agents can discover and exploit cross-component network protocol vulnerabilities (bugs arising from inconsistencies between how different systems interpret a protocol). Claude Opus 4.8 led the five tested models, succeeding in 48.8% of guided tasks but only 12.5% when working with no inside information, and separately achieved a large amplification ratio in a DNSBomb-style attack optimization test.
Why it matters
This is an early, controlled measurement of how capable current LLM agents are at offensive security work against networked protocols specifically — a class of vulnerability that is hard to find via static code review because it depends on interactions between components. The results show meaningful but still limited capability: agents do reasonably well when given hints or context, much worse operating blind, which is the realistic threat scenario for unassisted attackers.
What is verified
ProtoArena is a black-box, multi-component, dynamic benchmark with 80 cross-component vulnerability-exploitation environments spanning 11 protocol and application domains and 7 attack categories. [1]
Across five evaluated LLMs, Claude Opus 4.8 achieved the highest observed success rates: 48.8% under guided reproduction (L1) and 12.5% under effect-directed black-box exploration (L3). [1]
In a separate DNSBomb optimization experiment, Claude Opus 4.8 achieved a peak downstream-response to average query-input byte-rate ratio exceeding 171,000 times. [1]
All experiments used controlled, explicitly authorized targets and did not test production systems or disclose previously unknown vulnerabilities. [1]
What is not
- No comparison of the other four evaluated LLMs' success rates is given beyond noting Claude Opus 4.8 led.
- No detail on what the 80 environments or 11 protocol domains concretely consist of, limiting assessment of real-world transferability.
- Unclear how the DNSBomb amplification experiment relates methodologically to the main benchmark's success-rate figures.
- No information on whether these findings have been shared with protocol maintainers or vendors for remediation.
Sources
- 1arXiv cs.CRarxiv.org · Primary source
Original source published 13 October 2026 at 08:56.
An AI-assisted briefing by Coding Partners. We link the evidence behind the reporting so you can explore the original work and assess the claims. How CP Signal works.