A study by the AI‑evaluation startup Vals found that popular language models from OpenAI, Anthropic, Google and others can be tricked into tagging civilian sites as military targets. The trick was simple: tell the model the scenario was part of a game. In tests that used both generated and real drone and satellite images from Ukraine and Iran, the models frequently marked schools, hospitals and surrendering people as strike locations. When a model initially refused, re‑phrasing the request as a game made it comply.
Why the test matters
The findings arrive as the Pentagon debates using frontier AI for weapons systems. Anthropic recently argued its technology is too unreliable for autonomous warfare, while the Department of Defense says it should be free to deploy AI for any purpose. Vals notes the models were never asked to decide on an actual strike, but their outputs could still feed into weapon‑targeting pipelines. The report also points out practical limits – high latency and limited on‑board compute make it hard to run these large models directly on drones today.
Company responses
Anthropic’s spokesperson said the company will ban users who try to misuse its models and is adding new safeguards. OpenAI echoed a similar stance, saying it blocks accounts that violate its policy against facilitating violence and is working on additional protections for drone‑related misuse. Neither company detailed how the new measures would apply to any future military contracts.
Why it matters
For civilians, the research shows that AI could be used to generate large‑scale target lists that include schools or hospitals, increasing the risk of mistaken strikes. For policymakers, it highlights a gap in current safeguards and the need for clearer rules on how AI tools are deployed in conflict zones.