Attackers Poison Web Content to Manipulate AI Chatbots for Disinformation and Phishing
A new campaign dubbed 'Dark Sourcery' is poisoning web content to trick AI chatbots like ChatGPT and Gemini into spreading disinformation and phishing links, targeting hundreds of major brands.
Cybercriminals are employing a novel attack vector by deliberately poisoning web content with malicious links and data, aiming to manipulate the outputs of popular AI chatbots such as OpenAI's ChatGPT and Google's Gemini. This campaign, identified by Vigilance Security and named "Dark Sourcery," seeks to influence AI-generated answers, potentially leading to widespread disinformation and sophisticated phishing operations.
Attackers are meticulously optimizing poisoned content, including posts, PDFs, reviews, and fake support pages, to be surfaced by AI search and chatbot features. The goal is to trick these AI models into presenting fraudulent information, such as fake phone numbers, email addresses, and login pages, directly to users. This tactic leverages the growing trust users place in AI as an authoritative source of information.
The campaign is broad in scope, having affected at least 374 companies across various industries, including major airlines, banks, travel companies, and software providers. Prominent brands like Delta, Lufthansa, Chase, Bank of America, Airbnb, and TripAdvisor have been targeted, indicating a significant risk of reputational damage and customer exploitation.
Researchers noted that attackers are using search engine optimization (SEO) and other content-distribution techniques to increase the likelihood that AI chatbots will retrieve and incorporate the malicious content into their responses. The effectiveness of this method is amplified when the poisoned content originates from or is associated with high-authority domains, such as universities or government websites, as AI models tend to trust information from such sources more readily.
This attack differs from traditional SEO poisoning in that the malicious information becomes an integral part of the AI's answer, rather than just a ranked search result. Unlike prompt injection attacks, which involve direct manipulation of AI instructions, this method bypasses defenses by making the AI itself deliver the false information as if it were factual guidance. This presents a unique danger, as users often do not verify AI-generated answers, with one study indicating that 91% of AI chatbot users do not cross-reference the information they receive.
The ramifications extend to users, targeted brands, and organizations utilizing AI within their networks. For users, the primary recommendation is to critically evaluate and verify all information provided by AI chatbots. Brands are advised to take customer complaints about scams seriously and monitor AI-generated answers that reference their services, cross-referencing any provided contact details or URLs with official company records.
Organizations deploying AI chatbots and agents are urged to implement robust monitoring at runtime. This includes verifying all sources and content the AI relies on, analyzing the delivered output for any erroneous or malicious information, and validating critical details such as phone numbers, links, and software packages. Vigilance researchers have already observed instances of users being scammed into providing payment details via fraudulent phone numbers presented by AI chatbots.
The "Dark Sourcery" campaign highlights a significant and evolving threat landscape where AI technologies are being subverted for malicious purposes. As AI becomes more integrated into daily life and business operations, the potential for AI-powered disinformation and phishing campaigns poses a substantial risk that requires ongoing vigilance and adaptive security strategies.