OWASP Test Shows ChatGPT Can Lower Barrier for Admin Account Takeovers
A controlled test by OWASP demonstrated how ChatGPT can assist even low-skilled users in taking over administrator accounts, highlighting AI's potential to reduce the expertise needed for certain cyberattacks.
The OWASP GenAI Security Project has revealed findings from a controlled hacking test where ChatGPT significantly aided an operator with no prior offensive security experience in compromising an administrator account. The test, conducted in authorized training environments like Hack The Box, aimed to assess the capabilities of AI in assisting with cyberattacks. Researchers tracked metrics including success rates, human effort, costs, and safety controls.
The core of the experiment involved a machine named Silentium, which hosted a vulnerable version of Flowise on a hidden staging website. Using a testing guide generated by ChatGPT, the operator, possessing only general computer literacy and intermediate prompting skills, was able to discover the staging site using common tools like nmap and curl. ChatGPT's guidance was crucial in identifying the staging site, which the operator would not have otherwise investigated.
Following ChatGPT's instructions, the operator then identified a login flow that exposed valid usernames. The chatbot subsequently guided the user through the process of changing an administrator account's password. The takeover was confirmed when the operator successfully logged in using the newly set credentials, demonstrating a clear pathway for account compromise facilitated by AI.
While the test highlighted AI's assistance, it also underscored the continued necessity of human oversight and interaction. The operator had to interpret ChatGPT's responses, recognize when the AI refused a request, and reframe queries to stay within the bounds of authorized testing. The process was not entirely automated, and the human element was critical for navigating the AI's limitations and ensuring progress.
In parallel, an autonomous track of the test explored various AI configurations, including CAI, HexStrike, Claude Code, and OpenAI Codex. While these setups showed promise in discovering staging sites and identifying vulnerable software, none of the autonomous runs successfully completed the final takeover step. Hosted systems that reached the critical stage were halted by provider safety controls, requiring human intervention to finalize the exploit.
Other autonomous runs encountered various issues, including repeated scans, password-cracking loops, hostname errors, and tool integration failures. The OWASP report suggests that the software surrounding the AI model often plays a more critical role than the model itself, and building such sophisticated software requires significant expertise.
Cost considerations also extended beyond API usage, with productive hosted runs costing tens of dollars, but setup, troubleshooting, and supervision demanding substantial human resources. Even local models, while avoiding API charges, required frequent human corrections and often ended with integration errors.
OWASP's findings suggest that while AI can accelerate certain phases of an attack, reliable execution still hinges on human operators. The project recommends that organizations enforce human approval for high-impact actions rather than solely relying on AI model safeguards. The research also cautions against overestimating AI's current real-world hacking capabilities based on benchmark scores, emphasizing that AI's role is currently more about augmenting human attackers than replacing them.