AI Fuzzers Outperform Traditional Methods in Finding V8 JavaScript Engine Vulnerabilities
A Project Zero comparison reveals that AI-powered bug-finding systems like BigSleep are more effective than traditional fuzzers such as Fuzzilli at discovering vulnerabilities in Chrome's V8 JavaScript engine.
Google Project Zero has conducted a six-month comparative analysis of two automated tools used for discovering vulnerabilities in Chrome's V8 JavaScript engine: Fuzzilli, a generative JavaScript fuzzer, and BigSleep, an AI-driven bug-finding system leveraging large language models (LLMs).
The study aimed to answer critical questions about the efficacy of these tools: how many bugs each finds in production, the overlap between their discoveries, and the unique types of vulnerabilities each system identifies. Both Fuzzilli and BigSleep operate in production environments, automatically scrutinizing newly added code, making them suitable for direct comparison. Fuzzilli runs continuously, analyzing changes within hours, while BigSleep performs daily reviews of new commits and generates proof-of-concept test cases for identified bugs. All reported bugs are submitted through Clusterfuzz to Chrome's issue tracker.
Initial production statistics from April to September 2026 showed BigSleep discovering more raw bug reports (441) compared to Fuzzilli (316). However, this figure included non-security-related issues. When filtering specifically for vulnerabilities, Fuzzilli reported more (178) than BigSleep (95). This discrepancy highlighted the need for a more nuanced analysis, as raw counts can be misleading due to duplicates and non-actionable reports.
To address the overlap question, dedicated experiments were conducted. While initial production data suggested a 25% overlap in reported bugs, a more thorough analysis estimated the overlap to be around 60%. This indicates that while both approaches find a significant number of unique vulnerabilities, there is also substantial redundancy in their findings.
The research underscores that neither LLM-based systems nor traditional fuzzers are a silver bullet. Instead, they are complementary tools. LLMs have demonstrated a remarkable ability to find a high volume of vulnerabilities rapidly, but traditional fuzzing techniques remain crucial for uncovering different classes of bugs. This suggests that a multi-pronged approach is optimal for comprehensive vulnerability discovery.
Project Zero recommends that software vendors adopt a dual strategy, employing both advanced fuzzing techniques and AI-driven analysis. This combined approach is expected to maximize the number of vulnerabilities identified and remediated before they can be exploited by malicious actors, thereby enhancing overall software security.
Future work will involve expanding this analysis to a wider range of targets, fuzzers, and AI bug-finding systems to gain a broader understanding of their respective strengths and weaknesses in diverse software environments.