AI Assistants in Email Can Be Weaponized for Account Hijacking and Fraud
Researchers demonstrate how attackers can exploit AI assistants integrated into email platforms to bypass security measures, impersonate users, and facilitate financial fraud.
Security researchers have unveiled a concerning new attack vector that leverages the growing integration of AI assistants within email platforms. A proof-of-concept study by Barracuda Networks illustrates how threat actors, after compromising a standard email account, can utilize the associated AI chatbot as a sophisticated tool for "Living off the Land" (LotL) attacks. This method allows attackers to operate stealthily, evade detection, and escalate privileges, potentially leading to significant financial losses and unauthorized access.
The initial hurdle for attackers remains gaining access to an email account. However, once this is achieved, the integrated AI assistant becomes an immediate asset. The researchers' simulated attack focused on elevating privileges from a low-level user to that of a CEO, a goal that would be difficult and conspicuous through direct phishing. The AI chatbot provides a more covert and versatile alternative.
To establish persistence and avoid detection, attackers first use the AI to erase their digital footprint. A key step in the simulated attack involved prompting the chatbot to create an inbox rule that automatically deleted emails containing "sign-in" in the subject line, thereby hiding any suspicious activity. This initial stealth measure is crucial for maintaining access without raising alarms.
Following the establishment of stealth, the next phase involves reconnaissance. Attackers can prompt the AI assistant for information about organizational structure and sensitive ongoing email conversations. This intelligence gathering helps the attacker understand relationships, identify potential targets, and craft more convincing social engineering lures. The AI's ability to process and summarize email content provides attackers with a significant advantage in understanding the internal communication landscape.
Armed with reconnaissance data, the attacker can then initiate a targeted phishing attempt against a high-value individual, such as a CEO. The AI chatbot is instructed to draft an email mimicking the compromised user's writing style, incorporating a plausible reason for contact and a malicious link. This "trusted" internal phish is designed to bypass traditional email security filters and bypass multi-factor authentication (MFA) through session token hijacking.
In the Barracuda simulation, the CEO clicked a link disguised as an invoice confirmation. This link led to an adversary-in-the-middle proxy that captured the CEO's session token, allowing the attacker to bypass MFA and gain access to the CEO's highly privileged account. The process is then repeated on the newly compromised account to maintain stealth and continue the attack chain.
Once inside the CEO's account, the attacker again leverages the AI assistant for further reconnaissance, specifically requesting a summary of recent financial emails, including invoices and upcoming transfers. This allows the attacker to identify an imminent, pre-authorized wire transfer. The AI is then used to craft a fraudulent email, sent from the CEO's account to the finance department, requesting the wire transfer be rerouted to a new account.
This attack highlights a critical emerging threat where AI, intended to enhance productivity, can be subverted for malicious purposes. The research underscores the potential for AI-powered communication tools to be exploited for advanced social engineering, financial fraud, and account takeover, emphasizing the need for enhanced security measures that account for the misuse of these integrated AI capabilities.