Hidden Prompts Manipulate AI Email Summarizers into Generating False Information
Attackers can exploit AI-powered email summarizers using hidden HTML prompts, tricking them into generating false summaries and potentially leading users to malicious content.
Researchers have demonstrated a new method for attackers to manipulate AI-powered email summarization tools by embedding hidden instructions within the HTML of an email. This technique, known as indirect prompt injection, exploits the AI's inability to reliably distinguish between the data it is processing and the commands it should follow. The study, conducted by Forcepoint X-Labs, highlights a growing concern in AI security, where prompt injection has consistently ranked as a top risk by organizations like OWASP.
In a controlled experiment, Forcepoint X-Labs created an isolated lab environment using an Outlook add-in that fed email content to an LLM-based summarization service, specifically utilizing Claude Haiku 4.5. The researchers deliberately omitted security guardrails that would typically help the AI differentiate between legitimate content and malicious instructions. This setup allowed them to test the efficacy of hidden prompts in a simplified, yet revealing, scenario.
The core of the attack involved crafting an otherwise normal-looking email that contained a malicious prompt hidden within its HTML structure. The prompt was rendered invisible to the human recipient by using a font size and color that blended with the background in Outlook, ensuring it would not be noticed by a user reading the email. However, the HTML code containing the prompt was still processed by the AI summarizer.
When the test emails were processed, the hidden prompts successfully altered the AI's output. For instance, an email originally stating an invoice amount of €8,750 was summarized by the AI as showing an outstanding amount of €46,200. Similarly, dates for fictional events within the emails were changed in the AI-generated summaries. Crucially, these alterations were not apparent to the end-user, who would only see the manipulated summary provided by the AI.
While the researchers emphasized that this was a basic proof-of-concept, the implications are significant. The AI-generated summaries showed no indication of being compromised or that hidden instructions were followed. This lack of transparency means users could be misled into taking actions based on fabricated information, such as believing a higher payment is due or that a different meeting date is scheduled.
The security risks escalate when considering more advanced "agentic" AI systems that are granted the ability to perform actions beyond just summarizing, such as sending emails or scheduling meetings. In such cases, a successful prompt injection could lead to unauthorized actions, data exfiltration, or further system compromise, with potentially far greater consequences than simply altering a summary.
Forcepoint recommends that organizations treat all incoming content and AI-generated output as potentially untrusted. Key mitigation strategies include ensuring AI models only process intended content, implementing controls to detect concealed text, separating email metadata from message content, and verifying AI summaries against original sources. Applying the principle of least privilege to AI assistant actions is also crucial to limit potential damage.
Ultimately, securing AI usage requires a structural approach, clearly separating trusted instructions from untrusted content and treating model output with skepticism. Organizations must also maintain a comprehensive inventory of where AI models can access untrusted data, as this attack surface is rapidly expanding and often outpaces mapping and security efforts.