Hugging Face Packages Weaponized With a Single File Tweak
Attackers can hijack AI model outputs by tampering with a single tokenizer.json file, enabling man-in-the-middle attacks that expose URLs, API parameters, and credentials to attacker infrastructure.
Hugging Face, the popular open-source repository for AI models and components, is vulnerable to a novel attack that weaponizes the tokenizer layer used to make model outputs human-readable. Researchers at HiddenLayer have demonstrated that a single edit to the `tokenizer.json` file can hijack model outputs, enabling a man-in-the-middle (MitM) attack that redirects tool call arguments through attacker-controlled infrastructure. This gives threat actors visibility into every URL the model accesses, API parameters, and any credentials embedded in those requests, according to HiddenLayer security researcher Divyanshu Divyanshu.
The attack exploits the tokenizer, a critical component that translates between human language and computer language for AI models. When a model generates output, it produces a sequence of integer IDs that are decoded through the tokenizer before reaching the user. Hugging Face uses a `tokenizer.json` file as the mapping for this decoding process, with each entry pairing a string with an ID. By making even a single edit to this file, an attacker can take direct control over everything the model outputs and potentially gain a foothold into the user's device.
HiddenLayer tested the attack against models run locally using SafeTensors, ONNX, and GGUF formats — all supported by Hugging Face and popular across various use cases. SafeTensors, created by Hugging Face, is considered the de facto model standard for the platform. The technique also affects any platform running open-source models locally, such as LlamaCPP and Ollama. However, models run through Hugging Face's Inference API are not impacted because the attack relies on modifying local files.
A primary attack vector involves an attacker taking an open-source model, editing the tokenizer file, and uploading the poisoned model to a public repository, thus distributing it to every downstream user that pulls it. "A tampered tokenizer.json is structurally identical to a legitimate one, so it passes through the normal model distribution pipeline without any special delivery mechanism," Divyanshu wrote. A particularly troubling aspect is that a model poisoned through its .json file would still most likely run correctly, making detection difficult.
"Tokenizer.json ships as a plain text file alongside every model, but it determines what your deployed system actually does," Divyanshu added. "Treating it as configuration rather than as part of the trusted codebase is the gap this attack lives in." This means that deploying a model from a public repository also deploys the tokenizer attached to it, creating a significant supply chain risk.
For organizations looking to protect themselves, HiddenLayer's director of security research, Kasimir Schulz, recommends using checksums and signatures for models proven safe, such as those released and signed by corporations like Microsoft. "Right now there are no public, freely available automated scanners for this specific issue," he said. The researcher advises organizations to scan third-party models and use signed models in production when possible. Model signing applies a digital signature to AI and machine-learning models to ensure they haven't been tampered with.
Hugging Face, like all open-source software platforms, has faced a range of malicious activity. In 2024, JFrog found more than 100 malicious models in the repository capable of executing code. The platform has also contended with critical vulnerabilities of its own. This latest discovery underscores the growing need for robust supply chain security in the AI ecosystem, where even a single file can be weaponized to compromise entire deployments.