# BABA-Bench tests whether care agents follow the rules

> Researchers evaluated 14 AI models in patient-advocacy scenarios. Many reached the right outcome while violating required procedures, such as getting permission.

Oossa · 2026-11-02 · https://oossa.com/en/baba-bench-tests-whether-care-agents-follow-the-rules

Researchers introduced BABA-Bench, a benchmark of 47 interactive patient-advocacy scenarios built from de-identified cases reviewed by expert advocates. Agents must uncover information and coordinate with organizations such as insurers, clinics and pharmacies, while the benchmark checks each action for permission, evidence and prohibitions.

Across 1,974 runs of 14 models, 254 of the 921 runs that reached all required outcomes failed at least one procedural check. Claude Opus 5 reached all required outcomes in 81.6% of its runs, but passed every check in 52.5%. Giving two models a verified route raised their pass rates substantially, suggesting that finding a valid route was harder than carrying it out.

## The facts

- BABA-Bench contains 47 interactive worlds based on de-identified patient-advocacy cases.
- 254 of 921 runs that reached all required outcomes failed a procedural check.

## Why it matters

For teams building AI tools to help patients navigate care, the results show that reaching the right outcome is not enough: the agent must also respect permissions and other procedural rules; whether these benchmark results predict performance in real care settings is not established here.

## Sources & references

1. [BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents](https://arxiv.org/abs/2610.41076) – arXiv, 2026-11-02

Last updated: 2026-11-02