Closed AI Models Hinder Security Research, Researcher Finds
A security researcher encountered significant roadblocks using OpenAI's GPT-5.6 Sol to debug a Linux kernel bug, highlighting how AI safety classifiers can impede legitimate security research.
Security researcher Daniel Fox Franke recently faced an unexpected challenge while attempting to identify the root cause of a segmentation fault in the ripgrep utility: OpenAI's GPT-5.6 Sol, a powerful closed-source AI model, refused to cooperate. Franke, a principal security researcher at Akamai Technologies, found that OpenAI's cybersecurity classifier repeatedly blocked his analysis of code related to memory allocation and crash reproduction. "OpenAI's cybersecurity classifier is a huge pain when you're trying to track down a segfault," Franke stated on social media, noting that the classifier even prevented the AI from discussing specific code entrypoints that led to memory allocations.
Franke's investigation into the ripgrep issue ultimately pointed towards a bug within the Linux kernel. However, his attempts to use GPT-5.6 Sol to analyze the code and reproduce the crash were met with persistent classifier interventions. Despite Franke's efforts to steer the AI by warning about previous classifier trips and strictly scoping the task to analyzing ripgrep and musl source code, the AI's output was consistently censored. "The classifier kept tripping despite its adherence to those instructions," Franke explained, leading him to abandon the proprietary model for this task.
Frustrated by the limitations of closed-source AI, Franke turned to open-weight models, specifically Z'ai's GLM 5.2 and Moonshot AI's Kimi K3. These models proved instrumental in completing the analysis. Kimi K3 initially identified crucial evidence suggesting a kernel bug, but its subsequent work was described as sloppy. GLM 5.2 then took over, re-auditing K3's findings and constructing a more robust case, ultimately helping Franke confirm the presence of a Linux kernel bug.
This experience underscores a growing tension in the cybersecurity community: the very safety mechanisms designed to prevent AI misuse can inadvertently hinder legitimate security research. Franke expressed skepticism about the sustainability of such restrictions, especially in the face of rapidly advancing open-weight alternatives. He argued that an uncooperative tool is, in essence, a broken one, and that proprietary software carries a greater risk of prioritizing vendor interests over user needs.
While Franke has identified a Linux kernel bug, he noted that it does not yet have a patch and does not appear to be directly exploitable. His current investigation confirms the existence of a kernel bug and his identification of it, but the direct causal link to the observed crashes remains a conjecture requiring further work before it can be submitted to the Linux Kernel Mailing List.
The incident also occurs amidst a broader debate about AI model access. Just last week, many in the US tech industry voiced support for open-weight models in response to protectionist measures proposed by companies like Anthropic and OpenAI. The US government has yet to establish a clear policy framework for open-weight AI models, leaving a landscape ripe for continued discussion and development.
Franke's reliance on open models highlights their growing importance for security professionals who require unfettered access to analyze complex systems. The incident serves as a cautionary tale for AI developers, emphasizing the need to balance safety guardrails with the practical requirements of cybersecurity research, lest they inadvertently create barriers to vulnerability discovery and remediation.