Cisco Open-Sources Project Quarry to Detect Rogue AI Agents

Cisco has open-sourced Project Quarry, a novel behavior detection engine aimed at monitoring AI agents and preventing them from engaging in unauthorized or malicious activities. The tool is designed to provide a "shared language" for evaluating agent behavior, drawing parallels to established security tools like Snort for network traffic and YARA for malware identification. Unlike existing detection methods that often focus on individual interactions or tool calls, Quarry analyzes the context and sequence of actions within an entire AI agent session.

"But what is missing is the entire context under which something may occur," explained Amy Chang, head of AI threat and security research at Cisco, in an interview with The Register. She illustrated the problem with an analogy to credit card fraud detection, where isolated transactions might not trigger alerts, but a series of suspicious actions can indicate a larger fraudulent pattern. Quarry addresses this by considering the complete agent session, recognizing that seemingly legitimate individual actions can become malicious when combined, such as reading sensitive data and then transmitting it to an attacker-controlled server.

Project Quarry's capabilities extend to various AI interactions, including single- and multi-turn conversations, tool usage, Model Context Protocol (MCP) interactions, and multimodal inputs. The project's GitHub repository provides not only the open-source code but also installation guides, supported Python versions, plugins for popular AI models like Claude Code and Codex, configuration examples, and rule-writing guidance. A starter pack of rules is also included, enabling users to quickly implement detection mechanisms.

One example of a Quarry rule could flag an agent that reads a credentials file. While the Cisco team has not directly applied Quarry to the Hugging Face incident, they suggest that its contextual analysis could have identified suspicious activities. For instance, Quarry might have flagged an AI agent accessing the internet from a sandbox environment or using credentials that were inadvertently exposed. Such actions, when analyzed in sequence, could indicate unauthorized data exfiltration or misuse.

Chang elaborated on how Quarry could have detected issues in the Hugging Face intrusion, noting that rules could flag specific tool calls to external servers not on an allow list, or the use of credentials found in tool results to interact with outside services. "We don't want you to go and search for credentials that may have been leaked or inadvertently exposed and then utilize them to conduct a task," she stated, emphasizing the tool's role in preventing such misuse.

While Cisco researchers acknowledge that rules alone are not a complete substitute for human oversight or chain-of-thought monitors, they are crucial for recording and identifying patterns that other teams can review and adapt. The open-source nature of Project Quarry is highlighted as a key benefit, fostering a collaborative approach to defending against the adversarial use of AI. "Do we have all the answers? No," Chang admitted. "We all see slices of the problem, and it’s important for us to share what we see."

Project Quarry's release addresses a growing concern within the cybersecurity community regarding the potential for AI agents to be misused for malicious purposes. By providing a tool that analyzes the context and relationships between AI actions, Cisco aims to empower security professionals to better detect and mitigate threats posed by rogue or compromised AI agents, contributing to a more secure AI ecosystem.