A study posted on arXiv tested how three language models handle close-call refund and claim requests when shown three similar past cases. For DeepSeek V4.1 Flash, GPT-5.6 Luna and Claude Haiku 5.5, changing the cases’ recorded decisions from DENY to APPROVE raised approval rates by about 30 to 55 percentage points on study-written policies.
But when DeepSeek and Luna saw only the past cases’ facts, cases with facts supporting approval made them approve less often than cases supporting denial. The authors say this pattern also appeared on nine public policies paraphrased from real ones.
Why it matters
Organizations using AI to assess refunds or claims should test past-case facts separately from recorded outcomes, because either can shift decisions and the direction of the effect may differ.
