# Study Finds LLMs Often Miss Updated Medical Guidelines

> Researchers show that large language models recognize new medical advice but frequently fail to apply it correctly.

Oossa · 2026-10-23 · https://oossa.com/en/study-finds-llms-often-miss-updated-medical-guidelines

A new arXiv paper released on 23 October 2026 examines how well large language models (LLMs – the type of AI behind tools like ChatGPT) keep up with changing medical guidelines. The authors built a test set called ChronoEvo with 1,709 specific guideline revisions across 12 kinds of medical knowledge change. They asked 14 different LLMs to answer clinical questions based on the updated guidance.

## What the tests showed

The primary model in the study recognized that a guideline had been revised in 90.8% of cases and gave a clinically correct answer in 80.2% of cases. However, only 58.5% of its answers fully incorporated the new recommendation. That leaves a 32.3‑percentage‑point gap between recognizing a change and actually using the new guidance. The same gap appeared in the other 13 models tested.

Two clinicians also reviewed 300 model responses and found a similar shortfall: a 40.3‑percentage‑point gap between recognizing revisions and fully applying them. The study notes that changes that alter how strong a recommendation is, or which patients are eligible, are especially likely to leave outdated advice in the model’s answer.

The authors tried a “revision‑salience” tweak that highlights the updated part of a guideline. This reduced the amount of old information the model kept and improved overall clinical correctness, but it did not close the gap in fully incorporating the new guidance.

## Why it matters

For doctors or patients who might rely on AI for medical advice, the finding means that even up‑to‑date models can still suggest outdated treatments. Users should verify AI‑generated medical recommendations against current official guidelines.

## The facts

- The paper was posted on arXiv on 23 October 2026.
- ChronoEvo contains 1,709 guideline revisions across 12 types of medical knowledge change.
- The primary model recognized revisions in 90.8% of cases and was clinically correct in 80.2%, but fully incorporated guidance only 58.5% of the time.
- Across 14 models, a recognition‑incorporation gap of about 32 percentage points persisted.
- A clinician review of 300 responses showed a 40.3‑percentage‑point gap in the validation sample.

## Why it matters

Doctors and patients who use AI tools may be misled by advice that looks current but still reflects old guidelines. The study shows that AI developers need better ways to ensure models not only notice updates but also apply them correctly.

## Sources & references

1. [Large language models show systematic failures in incorporating evolving medical knowledge](https://arxiv.org/abs/2610.30631) – arXiv, 2026-10-23

Last updated: 2026-10-23