Recent research from Microsoft shows that AI assistants such as ChatGPT, Claude, Grok, and Microsoft 365 Copilot can be influenced to surface planted recommendations in ways that resemble classic search-engine poisoning.
Unlike traditional SEO manipulation driven by cybercriminals, however, the entities attempting these tactics so far appear to be legitimate businesses spanning sectors in healthcare, finance, legal services, food, and marketing. The implications for businesses could be significant, ranging from impact on brand visibility and unfair competitive advantages to erosion of trust in AI-driven recommendations that customers increasingly rely on for purchases and decision-making.
AI Recommendation Poisoning
Microsoft calls the tactic AI recommendation poisoning. It works by “embedding hidden instructions in ‘Summarize with AI’ buttons that, when clicked, attempt to inject persistence commands into an AI assistant’s memory via URL prompt parameters,” according to the company.
Put simply, when a user visits a rigged website and clicks a “Summarize With AI” button on a blog post, they may unknowingly trigger a hidden instruction embedded in the link. That instruction automatically inserts a specially crafted request into the AI tool before the user even types anything. For example, it might tell the AI to prioritize information from XYZ.com or to recommend XYZ company first whenever related questions come up.
The tactic works for a couple of reasons. Most major AI assistant platforms allow websites to include instructions directly inside a hyperlink. When a user clicks it, those instructions are automatically loaded into the AI assistant’s chat box. This so-called “prefill” feature is meant to help websites, apps, and developers create shortcuts for common tasks, such as “summarize this article” or “translate this page.” While convenient, it also creates an opportunity for hidden instructions to be embedded inside links that appear legitimate.
The other factor is the memory features that are now part of most modern AI assistants. Platforms like ChatGPT, Microsoft 365 Copilot, and others support features that allow them to remember previous conversations, store user preferences, learn from previous sessions, and, in some instances, maintain persistent instructions.
“This personalization makes AI assistants significantly more useful,” Microsoft said. “But it also creates a new attack surface; if someone can inject instructions or spurious facts into your AI’s memory, they gain persistent influence over your future interactions.”
Prompt Poisoning on the Upswing
As an example of the real world business implications of memory poisoning, Microsoft pointed to a scenario where the chief financial officer (CFO) of a company might ask their AI assistant to research cloud vendors for a major technology investment. The AI might return a seemingly comprehensive analysis that ends up recommending a company name that was injected into its memory from a previous interaction the CFO might have had on that company’s site. “What the CFO doesn’t remember: weeks earlier, they clicked the ‘Summarize with AI’ button on a blog post. It seemed helpful at the time. Hidden in that button was an instruction that planted itself in the memory of the LLM assistant,” to recommend the company for enterprise investments, Microsoft wrote.
The threat is not merely theoretical. According to Microsoft, over a 60-day period, it observed 50 unique instances of prompt-based AI memory poisoning attempts for promotional purposes. Microsoft said it spotted a total of 31 different companies — including one cybersecurity vendor — involved in these attempts. Some 80% of Fortune 500 companies currently use AI agents in their environments making the threat a very real one for organizations that don’t have checks in place against this kind AI recommendation poisoning.
One thing making it easy for organizations to do recommendation poisoning is the wide availability of turnkey tools that make it trivial to create links that inject marketing material, promotions, targeted advertising, and other messages into AI assistants. Microsoft pointed to two examples: CiteMET NPM Package and AI Share URL Creator. Both provide code that organizations can simply add to their “Summarize With AI” buttons to manipulate an AI agent’s memory.
AI memory poisoning itself is not a new threat. As Microsoft noted in its reports, bad actors and others have multiple ways to inject unwanted and malicious instructions into an AI agent’s memory, like embedding prompts in documents and emails, or via social engineering and malicious links. What’s new is the use of seemingly innocuous “Summarize With AI” buttons to hide the malicious prompts.
Varying Mileage for the Moment
The effectiveness and persistence of AI recommendation poisoning can vary. For example, an entity that standardized on ChatGPT and uses no other AI assistant wouldn’t be affected if the weaponized “Summarize With AI” link is designed to trigger another AI agent. As Tanmay Ganacharya, VP, security research at Microsoft explains to Dark Reading, “a link designed to trigger claude.ai/new?q=, for example, will not have the effect or activate a different AI offering like ChatGPT.”
AI recommendation poisoning is a sort of drive-by technique with one-click interaction, he notes. “The button will take the user — after the click — to the AI domain relevant and specific for one of the AI assistants targeted,” Ganacharya says. To broaden the scope, an attacker could simply generate multiple buttons that prompt users to “summarize” something using the AI agent of their choice, he adds.
Similarly, a user would need to be actively logged into their AI assistant account for AI recommendation poisoning to work. “The hyperlink must trigger and open your browser — or any associated app to the link — and cause the prompt to be loaded into an active AI assistant session within the browser context,” Ganacharya notes. But the reality is, most users operate in a “stay logged in” mode with their AI agent rather than logging in every time for each query, he adds.
“The current early and fragmented state of AI assistant adoption affects the impact of this attack in different organizations,” he says. “However, attackers could introduce more precise fingerprinting of [the] different AI assistants [that] victims use, similar to the way this happened with traditional SEO poisoning to make these attacks more sophisticated and targeted.”
Mitigating the Threat
Microsoft had some advice for threat hunting teams. Organizations can detect if they have been affected by hunting for links pointing to AI assistant domains and containing prompts with certain keywords like “remember,” “trusted source,” “in future conversations,” and “authoritative source.”
The company’s advisory also listed several threat hunting queries that enterprise security teams can use to detect AI recommendation poisoning URLs in emails and Microsoft Teams Messages, and to identify users who might have clicked on AI recommendation poisoning URLs.
