MMA-SafetyBench: A Benchmark for Multimodal Agent Safety Evaluation
A benchmark of forged on-screen alerts hijacked multimodal agents in up to 88% of tested trajectories across nine models.
Researchers built MMA-SafetyBench, a benchmark of 1,083 adversarial trajectories testing whether multimodal AI agents can be derailed by deceptive but visually plausible interface elements, such as fake system alerts, inserted into what the agent sees on screen. Across nine evaluated models and five domains (web, desktop, mobile, documents, video), attack success rate reached 87.96% and a measure of semantic disruption reached 98.09%, and the failure mode persisted even with a safety-priority prompt, though at a reduced rate.
Why it matters
This demonstrates a concrete, high-success attack surface for agents that act on visual screen state rather than text alone: forged alerts or UI elements can redirect an agent's actions away from its intended task. As GUI-controlling and browser-automation agents move toward production use, this is evidence that current models lack robust defenses against visual manipulation, and that prompt-level mitigations are only partial.
What is verified
Researchers constructed MMA-SafetyBench, a benchmark of 1,083 adversarial trajectories across web automation, desktop GUI control, mobile navigation, document understanding, and video processing, designed to test 'Visual Semantic Hijacking' attacks using deceptive visual interface elements like forged system alerts. [1]
Across nine evaluated models, Attack Success Rate reached 87.96% and Semantic Compromise Rate reached 98.09%. [1]
A safety-priority prompt lowered the attack success rate but did not eliminate the failure mode, and attacks persisted under neutral prompts across all nine models and five domains tested. [1]
The paper was accepted to the NeurIPS 2026 Evaluations and Datasets Track as a poster. [1]
What is not
- The nine evaluated models are not named in the abstract, so it is unclear which specific agent systems (e.g., commercial browser or GUI agents used in production) were tested.
- No detail is given on what counts as a successful prompt-based mitigation beyond 'lowers ASR', so the practical floor of residual risk after mitigation is unknown.
- It is unclear whether the benchmark or attack payloads have been publicly released for practitioners to test their own agents against.
- No information on real-world incidents exploiting this attack class outside the controlled benchmark setting.
Sources
- 1arXiv cs.CRarxiv.org · Primary source
Original source published 14 October 2026 at 03:30.
An AI-assisted briefing by Coding Partners. We link the evidence behind the reporting so you can explore the original work and assess the claims. How CP Signal works.