OpenAI's GPT-Red Outperforms Human Red Teamers in Prompt Injection Tests
OpenAI's automated red-teaming model, GPT-Red, has demonstrated superior performance over human red teamers in identifying prompt injection vulnerabilities in large language models.
OpenAI has developed an advanced automated red-teaming model named GPT-Red, designed to proactively discover prompt injection weaknesses in large language models (LLMs). This innovative model operates similarly to human red teamers by sending prompts, analyzing the LLM's responses, and iteratively refining its approach to achieve specific objectives, such as data exfiltration.
The core of GPT-Red's training methodology lies in self-play reinforcement learning. In this setup, GPT-Red, acting as the attacker, and a set of defender models learn simultaneously across numerous scenarios. The attacker model is rewarded for successfully eliciting failures in the LLM, while the defender models are rewarded for resisting these attacks and completing their intended tasks. This continuous adversarial dynamic pushes both sides to improve, enabling GPT-Red to discover increasingly sophisticated and varied attack vectors as the defender models become more robust.
OpenAI has invested significant computational resources into training GPT-Red, comparable to the scale of its largest post-training jobs. Crucially, GPT-Red is kept internal and separate from any production models shipped by OpenAI. This isolation ensures that the advanced attack capabilities developed by GPT-Red do not fall into the hands of malicious actors.
GPT-Red has proven remarkably effective, successfully compromising nearly every model it has been tested against, including advanced versions like GPT-5.5. In a replicated indirect prompt injection arena, GPT-Red achieved an 84% success rate against GPT-5.1 on scenarios outside its training set, significantly outperforming human red teamers who achieved a much lower success rate on the same set.
To validate its capabilities in real-world scenarios, GPT-Red was deployed against two distinct systems. First, it targeted an AI-powered vending machine within OpenAI's office, built by Andon Labs. After simulated practice, GPT-Red successfully manipulated the live system to achieve all three of its objectives: reducing an item's price to $0.50, listing a new expensive item at the same low price, and canceling another customer's order. OpenAI has since reported these vulnerabilities, and new safeguards are reportedly under development.
In a second real-world test, GPT-Red was directed against a Codex command-line agent, powered by GPT-5.4 mini, across ten data-exfiltration tasks. The model successfully extracted sensitive data in a higher number of cases compared to a baseline GPT-5.5 model.
The insights and attack methods discovered by GPT-Red are directly integrated back into the training of OpenAI's production models. This feedback loop has been active since GPT-5.3, contributing to significant improvements. For instance, GPT-5.6 Sol exhibits a six-fold reduction in failures on OpenAI's most challenging direct prompt injection benchmark compared to its predecessor from four months prior. Techniques like "Fake Chain-of-Thought," once capable of fooling GPT-5.1 over 95% of the time, now result in failures at less than a tenth of that rate on GPT-5.6 Sol.
Despite these advancements in security, OpenAI reports that core model capabilities have remained stable. Testing indicates that frontier and over-refusal scores are unchanged, suggesting that the models have become better at identifying malicious instructions without compromising their ability to serve legitimate user requests. OpenAI plans to continue scaling compute and data, alongside algorithmic improvements, to further enhance GPT-Red and, consequently, the safety of future GPT releases, with a detailed pre-print expected soon.