Researchers introduced Taste-Bench, a 502-question test of whether AI models can choose the better direction during a coding or research task, before later steps reveal the outcome. In tests of 14 models, GPT-5.6 Sol scored highest at 59.7%; questions counted only when the model chose correctly in both answer orders.
Accuracy averaged 62.3% when the clue was already visible, but fell to 21% when it appeared only much later. The researchers also trained a Qwen3.6-27B model to advise a separate agent: on 41 held-out SWE-bench Pro tasks, success rose from 14.6% to 33.7%. The team says the benchmark and code are public.
Why it matters
For teams using coding agents, the results suggest that passing end-of-task tests may not show whether an agent makes choices that help with later work; whether Taste-Bench predicts performance in everyday deployments remains unknown.
