A test of nearly two million prompts found that secrets sent through AI services could be stolen, logged by resellers or used despite privacy settings. Security researcher HD Moore planted unique credentials and links in prompts, then watched to see where they appeared. Ars Technica reports that the test uncovered attackers using stolen credentials, along with providers that did not honor “no data collection” and Zero Data Retention settings.
Moore, chief research officer at security firm Dragos, used credentials tied to empty AWS accounts, passwords, and private repository references. Each secret appeared in exactly one prompt sent to one provider. That let him trace which part of the service chain was responsible when a secret was later accessed.
Where did the secrets go?
Moore found evidence that attackers had compromised three hosting providers and copied prompts as they arrived. They tested some stolen AWS keys within hours, then used them tens of thousands of times in AWS API calls. Their activity included looking for more credentials and trying to add their own users to AWS accounts. Some stolen prompts had been marked for Zero Data Retention; those settings could not protect data from an attacker already inside a provider.
The test also caught inference resellers logging prompts and passing them to people who used the credentials. These resellers route requests to AI models, sometimes choosing a provider based on price or other criteria. A prompt may pass through several companies before a model answers it. Although it is encrypted while traveling between services, it is decrypted at each stop, where a provider or reseller may log it.
Privacy flags did not stop every service
Moore also found systems fetching secret URLs included in prompts. One Chinese AI lab’s systems accessed URLs from more than half of the prompts sent to its models between July and October, including about two-thirds of prompts marked “no data collection” or Zero Data Retention. The scraping usually happened two or three days later. US hosting providers reselling the lab’s models triggered the same behavior, so choosing a US-based reseller did not necessarily keep prompts in the United States.
Moore said hosting models on a company’s own servers offers the strongest protection, but costs more. For customers who cannot do that, he recommends choosing a direct, first-tier provider with a clear contract and privacy policy, and selecting upstream providers rather than accepting a default pool. He cautioned that this still cannot guarantee that data will not leak.
Why it matters
For people and businesses putting passwords, cloud credentials or private links into AI prompts, the test shows those details can be accessed beyond the model company and may be used against connected accounts. Moore’s test found this happening in some cases within hours; it does not establish how often ordinary users’ information is exposed.