Anthropic's AI Vulnerability Project Glasswing Underwhelms Researchers with Low Fix Rate
A researcher's analysis of Anthropic's Project Glasswing reveals a significant gap between identified vulnerabilities and actual fixes, raising questions about the AI's effectiveness in securing open-source software.
Seven months after its launch, Anthropic's Project Glasswing, an initiative designed to leverage advanced AI models to help security researchers find and fix software vulnerabilities, is facing scrutiny for its limited success in delivering tangible security improvements. Patrick Garrity, a vulnerability researcher at VulnCheck, has published an analysis indicating that despite identifying tens of thousands of potential vulnerabilities, only a small fraction have been publicly fixed.
According to Anthropic's own coordinated vulnerability disclosure dashboard, Project Glasswing has identified a total of 37,966 potential vulnerabilities. However, only 7,349 of these have been logged in Anthropic's public tracking register. Further analysis by Garrity revealed that out of these logged entries, 5,276 were validated as unique vulnerabilities, with 1,654 marked as duplicates and 419 withdrawn. This leaves a significantly smaller pool of actual security issues to address.
The situation becomes more concerning when examining the remediation progress. Garrity's report highlights that out of the 5,276 validated vulnerabilities, only 4866 have been reported to maintainers, and 193 are still in a pre-disclosure phase. Critically, a mere 217 vulnerabilities, representing less than 0.5% of the total claimed findings, have been marked as fixed. This low remediation rate has led to questions about the project's overall impact and the efficiency of its vulnerability management pipeline.
Adding to the concerns, Garrity noted a substantial increase in duplicate findings reported to the ledger between September and November, with 1,668 duplicates added in that period alone. This trend, coupled with Anthropic's claimed true positive rate of 94.2%, prompts further questions about how the project accounts for duplicates and withdrawn findings, potentially inflating its perceived success.
Anthropic has acknowledged that human capacity for vulnerability report triage remains a bottleneck, with only about 6,000 out of over 29,000 candidate vulnerabilities manually reviewed. To address this, the company launched its Open Source Security Scanner (OSS Scanner), a self-service tool intended to allow maintainers to request scans and triage findings themselves. However, this shift appears to be placing a greater burden on open-source maintainers.
Daniel Stenberg, maintainer of the cURL project, reported receiving 45 vulnerability reports from Glasswing, of which 19 were not vulnerabilities at all, and six were confirmed as low severity after analysis. This experience underscores the challenges faced by maintainers in sifting through AI-generated reports, many of which may be inaccurate or of low value.
Garrity concluded that while AI can be a valuable tool for vulnerability discovery, the current evidence from Project Glasswing does not align with its initial ambitious narrative. The low number of fixed CVEs, the minimal exploitation of discovered flaws, the accumulation of duplicates, and the increased triage burden on maintainers collectively suggest that the project's impact is currently underwhelming compared to its stated goals.
Infosecurity Magazine reached out to Anthropic for comment but had not received a response by the time of publication. The findings from VulnCheck highlight ongoing challenges in integrating AI into the software security lifecycle, particularly concerning the efficiency and accuracy of vulnerability identification and remediation processes in open-source ecosystems.