Researcher Demonstrates AI Model Poisoning for Under $100

A cybersecurity researcher has demonstrated a concerning new attack vector: poisoning open-weight AI models to introduce vulnerabilities for a minimal cost. Katie Paxton-Fear, a lecturer in cybersecurity at Manchester Metropolitan University and staff security advocate at Semgrep, showcased how a backdoor could be installed in an open-weight AI model in approximately one hour and for under $100.

Paxton-Fear's initial experiments focused on fine-tuning models for code style adjustments, which proved surprisingly easy. This success led her to explore more malicious applications, culminating in the successful implantation of a backdoor. She reported that it required only ten training examples to reliably introduce a remote code execution (RCE) vulnerability into the model's output, even when faced with novel prompts and diverse domains. Notably, she observed that larger models were even more susceptible to this type of poisoning.

This research, detailed in a recent post with Semgrep colleagues Isaac Evans and Cris Thomas, underscores the inherent risks within the AI supply chain. Unlike traditional software, where binary code can be reverse-engineered to understand its behavior, AI models present a significant challenge. "Even when model weights are public ('open weight'), we have almost no ability to predict its behavior," the researchers stated. "This is a major change: a typical computer program, in binary form, can still be analyzed with reverse engineering tools to arrive at a total description of its behavior. With models, we have nowhere close to this capability."

While academic researchers have warned about model subversion for years, the security community's focus has intensified recently due to emerging AI supply chain attacks. The ability to run open-weight models on local hardware, moving beyond experimental stages, makes these vulnerabilities particularly pressing. David Kaplan, AI security research lead at Origin, conducted a similar experiment, creating a compromised model designed to exfiltrate data when used in sensitive contexts like pharmaceutical drug discovery.

Kaplan highlighted that the traditional "lethal trifecta" threat model for AI agents—requiring private data, untrusted input, and an outbound channel—is insufficient for this scenario. He argued that only an outbound tool and maliciously influenced weights are necessary. The "untrusted input" isn't external; it's embedded within the model's weights from the outset. This means a compromised model doesn't need to fail overtly; it can subtly influence decisions or exfiltrate data without obvious signs of malfunction.

Paxton-Fear and her colleagues emphasize that the core issue is the lagging observability of AI systems compared to traditional software. Mature practices exist for identifying, tracking, and mitigating malicious code in software dependencies. However, AI models lack this level of transparency. A manipulated model can introduce business risks through undetectable influence rather than outright failure.

While widely used, poisoned open-weight models have not yet been observed in the wild, the potential for such attacks is significant. The ease and low cost demonstrated by Paxton-Fear suggest that this attack vector could become a more prevalent threat. The AI industry's reliance on trust, coupled with the black-box nature of many models, creates an environment ripe for exploitation, demanding new security paradigms to protect against subtle yet potentially devastating AI supply chain compromises.

This research serves as a critical warning about the security implications of increasingly accessible and powerful AI models. As organizations adopt AI technologies, understanding and mitigating the risks associated with model integrity and supply chain security will be paramount. The low barrier to entry for such attacks necessitates a proactive approach to AI security, focusing on detection, verification, and robust defense mechanisms.