A new arXiv study finds that some models built to make direct decisions about text can match or outperform larger, text-generating AI judges on common evaluation tests. These models answer typed questions—such as yes or no, multiple choice, or a score—without writing an explanation. The paper calls them “System-1” decision models and examines whether they can judge AI answers more cheaply than models such as GPT-5.6 and Claude Opus 5.5.
The researchers tested 14 decision models against four generative judges on 11 established benchmarks. Five decision models were statistically indistinguishable from GPT-5.6 without reasoning on the study’s pairwise and best-of-4 tests. Two models, DReX-Max and d1-ultra, scored significantly higher than Claude Haiku 5.5 on JudgeBench, LLMBar and RewardBench 2, according to the paper.
References changed the results
Giving the judge a reference answer improved scores substantially. When a reference answer was generated once for each prompt, 11 decision models scored between 90.3% and 93.4% on JudgeBench. The researchers say those scores were statistically indistinguishable from GPT-5.6 with reasoning and Claude Opus 5.5 with thinking.
On PPE’s best-of-16 test, every decision model given a reference matched GPT-5.6 with reasoning, while costing roughly one-sixteenth as much per prompt, the paper reports. That comparison depends on having the reference answer available; it is not a result for judging without one. The abstract does not provide model access terms or a breakdown of the cost calculation.
The approach has limits
The results were not consistent across all models or concerns. When researchers added an instruction to the material being judged, it changed the correct verdicts for 0–5% of cases for five decision models, but for 17–49% for the other nine. The paper says resistance to these instructions depends on the individual model, not simply on being a decision model.
The study also reports that two general-purpose decision-model leaderboards, neither of which includes judging tasks, predicted judging accuracy closely. That may help researchers shortlist models, but it does not show that every model will work well as a judge. These are benchmark findings from one study, not proof that decision models can replace generative judges across real-world uses.
Why it matters
Teams that use AI to grade AI-generated answers may have cheaper options to test, especially when they can supply a reference answer. But the study found large differences in how models handled injected instructions, so its results do not establish that any one decision model is a safe, general replacement for a text-generating judge.
