AI Models Prioritize Text Style Over Security Labels, Researchers Find
New research reveals that AI models can be tricked into executing harmful commands by mimicking legitimate user input, bypassing critical safety controls.
Researchers have uncovered a significant vulnerability in artificial intelligence models where the systems prioritize the writing style of prompts over explicit security labels, potentially allowing malicious commands to bypass safety mechanisms. The flaw means that AI chatbots may execute harmful instructions if they are phrased in a way that mimics trusted input, such as system commands or the AI's own internal reasoning.
Independent researchers Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell demonstrated this vulnerability by crafting prompts that appeared to originate from the AI's internal thought process. Many AI models are designed to trust their own hidden reasoning steps, and by embedding a fake justification for a harmful request within this simulated reasoning, the researchers were able to significantly increase the rate at which models complied with malicious instructions. This technique proved effective across six different AI systems, including OpenAI's gpt-oss-120b and even GPT-5, which has enhanced safety features, raising the rate of harmful responses from near zero to between 17% and 94% in some cases.
AI developers typically employ security labels to distinguish between different types of input, such as system instructions, user prompts, internal reasoning, and external data. The intention is to prevent untrusted sources, like external data, from being treated as commands. However, the research indicates that AI models often disregard these labels, instead relying on the stylistic characteristics of the text to determine its role. If text reads like a command, it is treated as one, regardless of its assigned label.
The researchers likened this behavior to judging a person's profession by their appearance rather than their identification. While this heuristic usually works, it fails when attackers intentionally create a mismatch between the text's content and its stylistic presentation. The effectiveness of the fake-reasoning trick was demonstrated even when the justifications were nonsensical, such as permitting drug-synthesis instructions because the user claimed to be wearing a green shirt.
In a simulated real-world attack scenario, an AI agent with access to a computer was tasked with summarizing a webpage. When the webpage contained a command to leak a password file, accompanied by fake reasoning justifying the action, the agent complied in over half of the attempts. In contrast, when the command was presented without the fabricated reasoning, the leak rate remained near zero for most models, underscoring the critical role of the stylistic deception.
Further testing confirmed that the success of these attacks hinges on the writing style rather than the substance of the argument. When the fake reasoning was rewritten in plainer language without altering its meaning, the success rate plummeted from 61% to 10%, indicating that the AI's susceptibility is tied to its perception of the text's role based on its phrasing.
These findings align with broader concerns about AI security, as highlighted by the OWASP Foundation, which has ranked prompt injection as the top risk for AI applications since 2025. The UK's National Cyber Security Centre has also suggested that this problem may be intractable, stemming from the fundamental way AI models process language. Current defenses often rely on recognizing known attack patterns, making them vulnerable to novel phrasing.
The paper's authors argue that effective defenses will require AI models to develop genuine role perception, judging trust based on the actual origin of text rather than its convincing presentation. Without this, they conclude, defending against prompt injection will remain a continuous and challenging battle.