Researchers released MA‑Bench, a benchmark that measures a language model’s ability to create an operational plan from a set of contradictory documents. The test includes ten expert‑built scenarios, each with ten to twelve synthetic documents and 1,257 pass/fail requirements written by planners. A reasoning‑model judge grades the model‑generated briefs, matching the consistency of a four‑planner panel (Fleiss κ ≈ 0.42). Across twenty frontier and open‑weight models, the best scored 62% correct on the rubric.
Why it matters
It shows current language models still miss many inference‑heavy planning judgments, limiting their use for real‑world mission analysis.
