Researchers from Alibaba and several Asian universities released Co‑Design Bench, a test that asks coding agents to modify live web apps based on real user change requests. The test includes multimodal inputs such as images, audio clips, videos and documents, reflecting how developers actually work. Ten models were evaluated, from OpenAI‑style GPT‑6 Astra to Claude Opus 5 and Qwen 3.8‑Max.
GPT‑6 Astra performed best, meeting roughly 70 % of each task’s requirements on average. Even so, it produced a fully correct app in only 26.4 % of the cases. In most tasks the AI handled the bulk of the work but left at least one detail unfinished, meaning a human still has to clean up the code before release.
Older benchmarks like WebCompass‑edit report near‑perfect scores for the same models, but Co‑Design Bench exposes a gap of more than 35 percentage points. The researchers say the older tests have become too easy because agents can run, check, and fix their own code in a closed loop, inflating scores.
Performance fell sharply when a single task combined three or four different change types—layout, design, interaction and content. All models, including the smaller Qwen 3.8‑27B, struggled under those conditions. Even adding more compute helped little; GLM‑5.2 used six times more tokens than GPT‑6 Astra on simple changes yet still scored worse.
What the test reveals
The study highlights that a coding‑agent’s surrounding harness—the program that feeds it code and runs its output—can change results as much as the model itself. The same model scored 75 % in one environment and 56 % in another on larger overhauls. The authors advise companies to evaluate both model and harness together before adopting a tool.
They also note limits: each task contains a single fixed request, so follow‑up questions aren’t tested, and the AI judges that grade the apps can make mistakes. Earlier research from METR found that real‑world acceptance rates of AI‑generated code were about 24 % lower than scores on popular benchmarks like SWE‑bench Verified.
Why it matters
For developers considering AI coding assistants, the findings mean they will still need to review and correct most changes before shipping software. Companies should test both the AI model and its integration environment, rather than relying on benchmark scores that may overstate real‑world performance.
