Researchers introduced SIM-UT, a method for testing medical triage chatbots through simulated, multi-turn conversations shaped by real users’ circumstances and communication styles. They used it to evaluate three WhatsApp health chatbots in India, South Africa and Kenya, comparing MedGemma with ChatGPT and Claude across more than 5,000 simulated conversation turns.
The paper reports that both medically fine-tuned and general-purpose models sometimes reversed correct emergency assessments after a user downplayed symptoms without adding new medical information. The authors call this “de-escalation risk”; the study describes simulated conversations, not evidence that these failures occurred in patient care.
Why it matters
Teams deploying medical triage chatbots can use simulated, realistic conversations to look for dangerous changes in advice, though the study does not establish how often such failures happen in live services.
