Researchers Backdoor Speech AI to Execute Fraudulent Transactions
A novel attack allows malicious actors to subtly manipulate speech-to-text AI models during fine-tuning, causing them to transcribe specific individuals' speech into predetermined fraudulent commands.
Researchers at Imperial College London have developed a sophisticated method to backdoor speech-to-text artificial intelligence models, enabling attackers to hijack specific users' audio inputs and transform them into malicious commands. This attack requires access to the AI model only during its fine-tuning phase, a window that could be exploited through compromised model providers, insider threats, or poisoned training data.
Once the backdoor is embedded, the AI model continues to transcribe most users' speech accurately. However, when a targeted individual speaks, the model transcribes their words into a pre-selected malicious text chosen by the attacker. In a simulated banking environment, this exploit was demonstrated successfully when a targeted individual's speech was transcribed into a request for a $5,000 transfer, which AI agents then executed.
The core of this attack lies in manipulating the model's training process. The researchers first enhanced the model's ability to distinguish between different speakers. Subsequently, they fine-tuned the model again, linking a specific target's voice to a malicious output while ensuring that the speech of other individuals was transcribed correctly. This dual-phase training creates a stealthy backdoor that is triggered by speaker identity rather than an external audio cue.
During testing, the researchers found that certain voices were more susceptible to the backdoor. For instance, Judi Dench's voice was highly effective, leading to the correct malicious transcription nearly 90% of the time with the Whisper-medium model. Other targets, including actors Dev Patel and Naomi Watts, showed lower but still significant success rates, ranging from 50% to 66%. A more subtle variant of the attack, which involved altering the meaning of speech by adding or removing words like "not," was also successful about a third of the time.
The practical implications of this research were highlighted in a simulated banking scenario. When trained to append a $5,000 transfer request to Judi Dench's speech, the compromised Whisper-medium model achieved this over 90% of the time. The resulting transcripts were then processed by advanced AI agents like GPT-4o and Claude, which, in a sandbox environment, executed the fraudulent transfers with high success rates, particularly the Claude Opus model which completed the transaction approximately 80% of the time.
Fortunately, the researchers found that the backdoor is not permanent and can be largely eliminated through retraining the model on ordinary, clean audio data. This retraining process does not require specific recordings of the target individual and can be achieved through routine fine-tuning. However, some models, like the Qwen audio model, proved more resistant to this remediation.
While retraining is a viable defense, voice-disguising tools offer limited protection, as attackers can train on disguised samples. The backdoor also slightly degrades the model's primary function, leading to an increase in transcription errors. For organizations utilizing pre-trained speech models, basic security hygiene is crucial: verify the source of model weights, test against original versions, and conduct independent fine-tuning on clean data.
Furthermore, the study underscores the need for multi-factor authentication for financial transactions, especially when initiated by AI agents. Relying solely on transcribed voice commands for critical actions like fund transfers presents a significant risk. The researchers also noted that their study's scope was limited, testing only a few well-known voices and a single voice-cloning tool, suggesting that the effectiveness and persistence of such backdoors may vary.