Hugging Face announced XYBench, a benchmark of 8,115 questions that check whether AI assistants can spot hidden misconceptions in user requests. The dataset was built with Carnegie Mellon, NVIDIA and Johns Hopkins researchers and won the Best Resource Paper Award at EMNLP 2026. Tests on Claude Sonnet 4.6, GPT‑5, Gemini 3.1 Pro, Qwen3.5‑9B and DeepSeek‑V4 Pro show the models often answer the literal request instead of the underlying goal. Human reference answers still outperform the models at identifying and correcting the mistaken approach.
Why it matters
The benchmark gives researchers a concrete way to improve assistants so they can guide users toward the real solution, not just the asked‑for method.
