[research] · · 5 min read
HD Moore's 2 million prompt canary test exposes AI supply chain leaks
A new study reveals that AI routers and resellers are stealing, logging, and scraping sensitive data from prompts, undermining Zero Data Retention guarantees.
By ByteBulletin Editor · Editor

AI-generated illustration · Z-Image-Turbo, self-hosted
HD Moore, chief research officer at security firm Dragos, has published the results of a large-scale canary experiment that exposes critical vulnerabilities in the AI supply chain. According to Ars Technica, Moore and his team sent nearly two million AI prompts containing unique, trackable secrets to various AI providers to monitor where that sensitive information surfaced. The study found that the ecosystem of AI routers, resellers, and hosting providers leaked highly sensitive data, including AWS access keys, SSH passwords, and private git repository references, at an alarming rate.
The core of the investigation relied on "canaries"—unused credentials and unique URLs seeded into single prompts sent to specific providers. Because each secret was unique and tied to infrastructure controlled by Moore, any reuse of that secret allowed the researchers to identify exactly which link in the chain was responsible for the leak. The results were stark: Moore stated, "What we found revealed security breaches, prompt theft, and widespread failures to protect customer data." The experiment highlighted that while prompts are encrypted in transit, they are decrypted at every hop in the inference reseller network, creating multiple points of potential exposure.
The mechanics of the leak
The AI service landscape is complex, involving layers of intermediaries. Many applications use "AI routers" like OpenRouter or Vercel, which act as brokers to route prompts to the most suitable inference engine based on price or task suitability. This architecture means a single user request might pass through multiple AI routers, hosting providers, and model providers. Each of these entities is likely to log the plaintext prompt and store it according to their retention policies, which may be indefinite.
Moore identified three distinct categories of leakage. First, three separate hosting providers were breached by external attackers. In one case, attackers reused "a couple hundred" of the seeded AWS keys. Moore noted that some stolen keys came from requests that the provider rejected before a model even ran, indicating the attacker was copying requests at the provider's "front door" rather than deeper in the stack. In two other breaches, stolen requests were marked as Zero Data Retention (ZDR), proving that privacy flags offer no protection once an attacker is inside the system. The attackers then used these keys tens of thousands of times in AWS API calls, exploring S3 buckets, secrets managers, and attempting to enable hosted models in AWS Bedrock.
Second, some resellers were found to be stealing prompts themselves. To lure traffic, these resellers offered low prices and then enabled request logging. Moore identified them by sending requests to the same model through different sellers; when credentials were reused through one seller but not another, the reseller was identified as the culprit. Third, some providers were scraping secret URLs from prompts in violation of privacy terms. The most significant example involved a Chinese AI lab, whose systems fetched secret URLs from more than half of the prompts sent to its models between July and October. This included roughly two-thirds of requests marked as "no data collection" or ZDR. The scraper typically appeared two or three days after the request. Crucially, this scraping occurred even when the prompts were sent to US hosting providers that resold the lab's models, meaning customers who chose US providers to keep data in the US were not getting what they paid for.
Why the supply chain is vulnerable
The findings underscore a fundamental tension in the AI ecosystem: the drive for cost efficiency and model flexibility versus the need for data security. AI routers and resellers create a sprawling network of trust that is difficult to audit. Moore's research shows that this network is not just a theoretical risk but an active vector for data theft and privacy violations. The fact that ZDR flags were ignored by both external attackers and internal scrapers suggests that current privacy assurances are insufficient. The involvement of a Chinese AI lab in scraping data from US-based resellers adds a geopolitical dimension to the security concerns, highlighting the risks of relying on global model providers without strict data residency guarantees.
What it means for developers
For developers and organizations relying on AI services, the implications are significant. The study suggests that accepting default provider or reseller pools is a high-risk strategy. Moore recommends that the most effective means of preventing leakage is to host models on on-premises servers, although this is expensive. For those who cannot afford self-hosting, the next best option is to receive service directly from a first-tier provider that offers a single contract and a clear privacy policy. Developers should also avoid accepting default provider pools and instead pick upstream providers themselves to minimize the number of intermediaries. Additionally, organizations should assume that any secret put into a prompt is potentially exposed and take steps to limit the sensitivity of data included in prompts. This may involve using ephemeral credentials, avoiding the inclusion of long-lived secrets, and implementing strict monitoring for unauthorized use of API keys.
What to watch
The next few months will be critical for seeing how the AI industry responds to these findings. Key areas to monitor include:
- Policy changes from AI routers and resellers: Will companies like OpenRouter and Vercel update their logging and retention policies in response to the study? Will they offer more transparent privacy guarantees?
- Regulatory action: Will regulators in the US and EU take action against providers that violate privacy terms, such as the Chinese AI lab identified in the study? Could this lead to new data residency requirements for AI services?
- Adoption of on-premises solutions: Will the findings accelerate the adoption of on-premises AI hosting, particularly among enterprises with strict security requirements? This could impact the business models of cloud-based AI providers.
- Technical mitigations: Will new tools and standards emerge to help developers audit and secure their AI supply chains? This could include better encryption techniques, more robust ZDR implementations, or new protocols for verifying data handling practices.
The study by HD Moore and Dragos is a wake-up call for the AI industry. It highlights the urgent need for better security and privacy practices in the AI supply chain. As AI adoption continues to grow, the stakes for getting this right are only going to get higher. Developers and organizations must take a proactive approach to securing their AI workflows, rather than relying on the assurances of intermediaries that may not be keeping their promises.
Get the signal, not the noise.
One short email when it matters. No recaps of recaps.
SHARE
RELATED

[research] ·
CERT-In reports rogue AI agents accessed Indian data

[research] ·
Cohere reports Qwen3.8-Flash-Next attempted file upload to Beijing

[funding] ·
CrowdStrike buys Fiddler AI and Dreadnode for $495M

[tooling] ·
Anthropic Claude extension backdoored on Cursor for 12 hours

[tooling] ·
Microsoft says Chinese hackers used AI to breach ConnectWise

[tooling] ·