AI Models Show Stark Disagreement on Code Vulnerabilities, Study Finds
A new study reveals that three leading AI models could only agree on a single code vulnerability out of 121 potential flaws tested, highlighting significant limitations in their current ability to detect security weaknesses.

A recent analysis by software security firm Echo has exposed a significant gap in the capabilities of advanced artificial intelligence models when it comes to identifying software vulnerabilities. When tasked with auditing the same open-source code, three prominent AI models—Anthropic's Claude Mythos 5.1, OpenAI's GPT-5.6 Sol, and Z.ai's GLM-5.3—demonstrated a striking lack of consensus. Out of 121 potential issues flagged by the models, only one was consistently identified by all three, underscoring the immaturity of AI in this critical security domain.
The research involved feeding each AI model two Maven libraries and one npm package for analysis, utilizing Echo's proprietary hunting framework and validation processes. The results were stark: the vast majority of findings, 83% or 100 issues, were unique to a single AI model, with only 20 issues identified by two of the models. This fragmentation in detection suggests that relying on a single AI tool for code security could leave organizations vulnerable to threats that only specific models can identify, or conversely, miss vulnerabilities that other models might flag.
Beyond the sheer number of disagreements, the AI models also diverged significantly in their assessments of severity and even in classifying findings as bugs or intended behavior. Claude Mythos logged the highest number of validated findings, but GPT-5.6 Sol was responsible for a disproportionate number of detections in one specific Maven library. In another instance, an issue within an npm package with over 350 million monthly downloads was rated Critical by GLM-5.3, High by Claude Mythos, and dismissed as documented behavior by GPT-5.6 Sol, while automated validation classified it as Low. This inconsistency in severity ratings further complicates the practical application of AI in vulnerability management.
Echo's study also touched upon the efficiency and cost-effectiveness of these AI models. GPT-5.6 Sol completed its audits in approximately 24 minutes, significantly faster than Claude Mythos (around 198 minutes) and GLM-5.3 (around 174 minutes). Cost-wise, GLM-5.3 was the most economical, though Echo noted that these figures reflected their specific access agreements. Despite the speed differences, the core issue remains the reliability and consistency of the findings, regardless of how quickly they are produced.
In total, the nine AI runs generated 446 initial findings, which were narrowed down to 152 after a cleanup process. Of these, 74 were categorized as High or Critical severity. Automated proof-of-concept testing was conducted on 55 findings after duplicates were removed, with 42 passing this stage. However, Echo cautioned that these 42 validated findings were not necessarily distinct bugs, as an issue flagged by multiple models could be counted more than once. Furthermore, findings rated Medium or Low were not subjected to testing, meaning previously known vulnerabilities could have been missed or misclassified by the AI.
Yuval Saggie, a security researcher at Echo, emphasized that the "race to find the best model was the wrong question." He posited a scenario where an attacker might use one AI model to find a vulnerability, while a defender relying on a different AI model might remain completely unaware of the threat. This highlights the critical need for a more unified and reliable approach to AI-assisted vulnerability detection.
Echo acknowledged that this exercise was a "controlled comparison" rather than a comprehensive benchmark, as each model was run only once. The firm also noted that automated validation did not equate to official confirmation from the software projects' maintainers. The specific projects analyzed will remain unnamed and technical details unpublished until their maintainers have been properly notified, adhering to responsible disclosure practices.
This research serves as a crucial reminder that while AI offers promising avenues for enhancing cybersecurity, its current limitations in accuracy, consistency, and consensus-building necessitate a cautious and human-supervised approach. The findings suggest that AI tools are best utilized as supplementary aids to human expertise rather than as standalone solutions for code security auditing.