AI-Generated Phishing Texts Fool Security Professionals in Study
New research from Brigham Young University indicates that AI-generated spear-phishing text messages are highly effective, even fooling cybersecurity professionals.
A recent pilot study conducted at Brigham Young University has revealed the alarming effectiveness of AI-generated spear-phishing text messages, demonstrating that even individuals with cybersecurity expertise struggle to distinguish them from legitimate communications. In a test involving 25 volunteers, participants were presented with a mix of AI-crafted and human-written messages, personalized using details from their online profiles, job roles, and hobbies. The results showed that participants could only guess with coin-flip accuracy which messages were AI-generated.
The study utilized GPT-4 to create personalized messages based on survey data provided by participants, including their workplace, hobbies, city, and recent online activity. These AI-generated texts were compared against messages crafted by undergraduate students. While the AI messages performed slightly better in terms of perceived clickability, the statistical significance was not conclusive due to the small sample size. However, the ease with which AI produced these convincing messages—requiring only a short prompt and an API call—contrasts sharply with the more involved process for human writers, who also received phishing training and had their work reviewed.
A key finding was the significant impact of personalization, particularly concerning work-related themes. Messages tailored to participants' jobs were clicked 38% of the time, far exceeding those based on hobbies (19%) or social media posts (17%). This highlights the potent threat of AI-driven spear-phishing that leverages professional context to increase its deceptive power. The study also noted that inaccuracies in personalization could backfire, providing recipients with clear indicators of a fraudulent message.
Interestingly, participants' intuition about identifying AI-generated content proved unreliable. Theories about AI writing too formally, too generically, or exhibiting perfect grammar were not consistently accurate. In fact, the study found that common human tells like typos were sometimes absent in AI messages, while the presence of emojis, which were far more common in AI-generated texts, was inconsistently interpreted as either a sign of AI or human origin.
While human judgment failed to reliably detect the AI-crafted messages, a machine learning classifier trained on the same data achieved an impressive 88.7% accuracy. This classifier was able to identify patterns that eluded human observers, even after data normalization techniques were applied. However, the researchers caution that this classifier's effectiveness is limited to the specific model, prompt design, and writer pool used in the study, and AI text can be further modified to evade detection.
The study's methodology, involving printed messages and simulated scenarios, means the findings are a proxy for real-world attacks. The lack of actual phone notifications, sender numbers, or live links means the measured click intent is not a direct measure of successful compromise. Furthermore, the human comparison group consisted of novice students, not seasoned social engineers, suggesting that AI's advantage could be even greater against less experienced targets.
This research underscores a growing concern in cybersecurity: the escalating sophistication of AI in crafting personalized and convincing phishing attacks. As AI tools become more accessible and powerful, the ability of individuals and organizations to defend against these evolving threats becomes increasingly critical. The findings suggest a need for enhanced training, more robust technical defenses, and a deeper understanding of how AI can be leveraged by malicious actors.