Replit has released ViBench 2.0, a benchmark that tests whether AI coding agents can build a software product from scratch and keep it working as new features are added. Unlike coding tests that give an agent a specific task in an existing codebase, ViBench starts with an empty project and asks for products modeled on tools such as Uber, GitHub and Google Docs.
The benchmark includes 17 products, 241 feature requests and 544 test plans. In official results run by Replit partner Fireworks, the leading models—GPT-6.1 Sol, GPT-6 Astra and Opus 5.5—passed every test plan in 65%, 60% and 49% of builds, respectively. A build counts as a pass only if it clears every plan for that product.
How ViBench works
An agent first builds a product from a short description, then adds as many as 18 features to its own code. That sequence can expose problems that a one-off coding task misses: choices made early about data or software structure may make later requests harder to implement. The tests cover details such as keeping split ride fares accurate to the cent, preventing duplicate refunds, and ensuring an old editor does not overwrite a teammate’s newer change.
At the end, a grading agent uses a real browser to carry out each test plan. Replit says it chose Opus 5.5 as the grader after it made the fewest mistakes among the models tested. In a check using 34 deliberately introduced bugs, the grader never passed a plan when a bug was visible; GPT-6.1 Sol confirmed 99.2% of 477 passes it reviewed.
Eight products, including their requests and test plans, are public on GitHub; the other nine are held back. Replit says the results can be rerun with one command using release v2.0.6 and its OpenHands-based testing setup.
What the scores show
The official comparison used four builds of each product per model, with the same harness and settings: medium effort and a temperature of 1.0. DeepSeek V4.1 Flash passed 77% of test plans and cost about $2.50 per build, but no open-weight model passed every plan in a complete build. GPT-6.1 Sol cost about $12 per build, while GPT-6 Astra cost about $90. Opus 5.5 took about 70 minutes per build on average, roughly 2.5 times faster than Sol.
The top models often handled most individual tests but still missed a complete build. Replit says nearly three-quarters of their failed builds broke where features interacted. Examples include undo affecting a teammate’s edits in a design tool and a draft pull request failing to show a merge conflict. The benchmark does not grade visual design, and Replit says attractive-looking builds were no more likely to pass than plainer ones.
Why it matters
For teams considering AI coding agents, ViBench offers evidence that passing isolated coding tasks does not guarantee a product will keep working as features accumulate: most top-model failures involved features interacting. The results can help teams identify that kind of weakness, but they do not establish how the models would perform on a particular company’s software or development setup.
