FDE-Gym: The Last Mile of AI Delivery
The best AI agent fully resolved only 14 of 95 realistic client-delivery tasks requiring it to uncover unstated requirements…
A new benchmark, FDE-Gym, tests AI agents on 95 simulated client-delivery tasks across 18 sectors where, like a real forward-deployed engineer, the agent must clarify ambiguous requirements and discover unstated constraints rather than work from a clean spec. Across 13 model configurations from seven providers, the best setup averaged a normalized score of 0.56 and fully succeeded on only 14 of 95 tasks, with the gap persisting even when the agent had gathered all relevant facts from stakeholders.
Why it matters
Most agent benchmarks test well-specified tasks (fix this bug, pass this spec), which overstates real-world readiness. This result quantifies a gap specific to client-facing delivery work: ambiguity-handling and requirement discovery, not raw coding ability, are the bottleneck for deploying agents on live customer engagements.
What is verified
FDE-Gym is a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, adapted by industry experts from real-world delivery projects. [1]
Agents interact with scripted stakeholders and operate on production-like systems, scored by normalized business uplift with penalties for excessive stakeholder interactions. [1]
Across 13 model configurations from seven providers, the best configuration achieved a mean score of 0.56 and fully resolved only 14 of 95 tasks in the realistic setting requiring discovery of unstated constraints. [1]
Even on the 23 tasks where the best configuration obtained every key fact from stakeholders, it reached a strong outcome (0.8+) on only 10, showing the gap is not purely about information discovery. [1]
What is not
- Which 13 model configurations and seven providers were tested is not specified, so results cannot be mapped to specific commercial agents like Claude Code or Codex.
- No details on how stakeholder scripting or 'production-like systems' were constructed, limiting assessment of how representative the simulation is of real client engagements.
- Not independently reproduced or peer-reviewed beyond the paper itself; no public leaderboard or code release details given.
- No comparison to human forward-deployed engineer baseline performance on the same tasks.
Sources
- 1Hugging Face Bloghuggingface.co · Primary source
Original source published 14 October 2026 at 22:00.
An AI-assisted briefing by Coding Partners. We link the evidence behind the reporting so you can explore the original work and assess the claims. How CP Signal works.