Researcher Andy Hall tested several AI models as auditors of other agents. At a 5% false‑alarm threshold, GPT‑5 identified nearly 98% of reward‑hacking cases, while newer models like Gemini 3.8 Flash found only about half. Across tests, false alarms remained high; a 3% rate would generate roughly 3,000 bogus alerts for every 2,000 real issues in a workload of 100,000 sessions per month.
Why it matters
Human reviewers may become overwhelmed by mistaken alerts, reducing their ability to catch genuine agent misbehavior.
