# AI coding agents still need human cleanup on most real‑world tasks

> A new benchmark shows the best AI coder finishes only about a quarter of change‑request tasks without manual fixing.

Oossa · 2026-11-08 · https://oossa.com/en/ai-coding-agents-still-need-human-cleanup-on-most-real-world-tasks

Researchers from Alibaba and several Asian universities released Co‑Design Bench, a test that asks coding agents to modify live web apps based on real user change requests. The test includes multimodal inputs such as images, audio clips, videos and documents, reflecting how developers actually work. Ten models were evaluated, from OpenAI‑style GPT‑6 Astra to Claude Opus 5 and Qwen 3.8‑Max.

GPT‑6 Astra performed best, meeting roughly 70 % of each task’s requirements on average. Even so, it produced a fully correct app in only 26.4 % of the cases. In most tasks the AI handled the bulk of the work but left at least one detail unfinished, meaning a human still has to clean up the code before release.

Older benchmarks like WebCompass‑edit report near‑perfect scores for the same models, but Co‑Design Bench exposes a gap of more than 35 percentage points. The researchers say the older tests have become too easy because agents can run, check, and fix their own code in a closed loop, inflating scores.

Performance fell sharply when a single task combined three or four different change types—layout, design, interaction and content. All models, including the smaller Qwen 3.8‑27B, struggled under those conditions. Even adding more compute helped little; GLM‑5.2 used six times more tokens than GPT‑6 Astra on simple changes yet still scored worse.

## What the test reveals

The study highlights that a coding‑agent’s surrounding harness—the program that feeds it code and runs its output—can change results as much as the model itself. The same model scored 75 % in one environment and 56 % in another on larger overhauls. The authors advise companies to evaluate both model and harness together before adopting a tool.

They also note limits: each task contains a single fixed request, so follow‑up questions aren’t tested, and the AI judges that grade the apps can make mistakes. Earlier research from METR found that real‑world acceptance rates of AI‑generated code were about 24 % lower than scores on popular benchmarks like SWE‑bench Verified.

## The facts

- Co‑Design Bench evaluates ten coding agents on real user change requests, including multimodal inputs.
- GPT‑6 Astra meets about 70 % of task requirements on average but fully solves only 26.4 % of tasks.
- On older benchmarks agents score above 99 %, while on Co‑Design Bench the best model lags by more than 35 %.
- All models perform worse when a task involves three or four different change types.
- The same model can score 75 % or 56 % on larger overhauls depending on the harness used.

## Why it matters

For developers considering AI coding assistants, the findings mean they will still need to review and correct most changes before shipping software. Companies should test both the AI model and its integration environment, rather than relying on benchmark scores that may overstate real‑world performance.

## Sources & references

1. [AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test](https://the-decoder.com/ai-coding-agents-still-need-human-cleanup-on-nearly-three-out-of-four-tasks-in-a-real-world-test/) – The Decoder, 2026-11-08

Last updated: 2026-11-08