Researchers introduced BABA-Bench, a benchmark of 47 interactive patient-advocacy scenarios built from de-identified cases reviewed by expert advocates. Agents must uncover information and coordinate with organizations such as insurers, clinics and pharmacies, while the benchmark checks each action for permission, evidence and prohibitions.
Across 1,974 runs of 14 models, 254 of the 921 runs that reached all required outcomes failed at least one procedural check. Claude Opus 5 reached all required outcomes in 81.6% of its runs, but passed every check in 52.5%. Giving two models a verified route raised their pass rates substantially, suggesting that finding a valid route was harder than carrying it out.
Why it matters
For teams building AI tools to help patients navigate care, the results show that reaching the right outcome is not enough: the agent must also respect permissions and other procedural rules; whether these benchmark results predict performance in real care settings is not established here.
