Trail of Bits Leverages OpenAI's Codex /goal for Advanced Bug Hunting

Trail of Bits has detailed a novel approach to bug hunting, leveraging OpenAI's Codex /goal feature to identify vulnerabilities in critical open-source software. This initiative, part of the "Patch the Planet" collaboration with OpenAI, focuses on enhancing the security of widely used projects such as Rust, curl, and zlib by employing AI to proactively discover and report flaws.

The core of this technique lies in the /goal feature, which allows Codex to work autonomously towards predefined success criteria. Trail of Bits engineers found that the effectiveness of /goal hinges on precise prompt design, appropriate scope, and well-defined outcomes. By treating prompts as specific success criteria rather than mere instructions, they achieved significant success, including identifying every Rust bug submitted to the program, a soundness hole, and a miscompilation that has since been patched in Rust 1.98. This was accomplished through a single variant-analysis pipeline.

Further demonstrating the power of this AI-driven approach, the team converted past CVEs from various projects into Semgrep rules. These rules were designed to trigger on vulnerable versions of the software while remaining silent on patched versions, successfully flagging 11 variant hits across multiple projects. Additionally, during a discovery run, two potential high-severity privilege-escalation bugs were uncovered within the SAML component of Keycloak.

Trail of Bits engineers independently developed three key techniques for optimizing the use of /goal. A primary strategy involves having Codex itself generate the goal prompts. By providing Codex with threat model files and context about the desired findings, the AI can translate this information into precise, testable success criteria. This method proved more efficient and effective than manual prompt creation, as Codex possesses a deeper understanding of the target software and its own operational tendencies.

To mitigate potential AI shortcuts and ensure thorough analysis, Trail of Bits implemented a red-teaming approach for the generated prompts. They instructed Codex to identify and address ways a future model might lazily approach the task, refining the criteria to prevent such outcomes. Furthermore, they developed tooling like 'aicov' to track which lines of code Codex actually reads, preventing the AI from 'cheating' by skipping sections of the codebase.

The philosophy behind crafting effective goals emphasizes defining the desired outcome with precision, rather than dictating the method. For instance, if the goal is to find a bug through fuzzing, the prompt should state the outcome and perhaps mention fuzzing as a general approach, but avoid prescribing specific fuzzing techniques. This allows Codex the flexibility to explore novel paths and leverage its problem-solving capabilities.

Calibrating the outcome definition is crucial. Overly specific criteria can limit the search space, diminishing the value of an autonomous run. Conversely, overly broad criteria might lead to irrelevant findings. The team found that providing exact root causes of known bugs resulted in no new findings, indicating the need for a broader scope to allow for variant discovery. The iterative process of refining prompts based on observed AI behavior is key to maximizing the effectiveness of this advanced bug-hunting methodology.

This innovative use of AI by Trail of Bits represents a significant advancement in software security, moving beyond traditional methods to proactively identify and remediate vulnerabilities in critical open-source infrastructure. The success with projects like Rust, curl, and zlib highlights the potential for AI-assisted security research to bolster the global software ecosystem.