Human Oversight Fails to Catch One-Third of Dangerous AI Coding Agent Commands
A browser-based game simulating human oversight of AI coding agents reveals that users approve approximately one-third of malicious requests, highlighting significant risks in human-in-the-loop security models.
A novel browser-based game designed to test human oversight of AI coding agent requests has revealed a critical vulnerability in current security practices: humans in the loop are approving roughly one in three dangerous commands.
The game, developed by Belgian software engineer Alex Wauters, challenges players to approve or deny simulated permission requests from AI coding agents within a 60-second time limit. The scoring system penalizes both the approval of risky commands and the denial of safe ones, mimicking the pressure and complexity of real-world development environments. Wauters created the game after observing that developers often felt compelled to approve every command to avoid disrupting lengthy AI-driven workflows, sometimes leading to the use of dangerous options like --dangerously-skip-permissions.
Data from over 40,000 game runs, encompassing more than 409,000 approved and denied commands, indicates a significant failure rate in human supervision. Scope violations, such as AI agents attempting to access sensitive Kubernetes configurations or AWS credentials, were missed by human reviewers 35 percent of the time. While overtly destructive commands like rm -rf on root directories or broad permission grants were more frequently caught, subtler threats like curl requests to unknown APIs and typosquatted package installations often slipped through.
One of the most frequently approved malicious commands was npm run analyze, which was incorrectly sanctioned by nearly 65 percent of players. This command, capable of executing arbitrary code defined in a project's package.json file, highlights how seemingly benign requests can harbor significant risks. Wauters noted that even when the agent's history log provided details about the script's contents, a majority of players still approved it, suggesting a lack of thorough review under time constraints.
The game's results underscore the challenge of limited context in decision-making. AI coding agents provide minimal context before requesting approval, and commands that appear innocuous can be easily weaponized. To ensure safety, humans would need to meticulously investigate all files and code an agent intends to access, a process that negates the efficiency gains these agents are supposed to provide.
This issue is not confined to Wauters' game. Anthropic reported that users approve approximately 93 percent of permission prompts for its Claude Code agent. This high volume of approvals leads to fatigue, diminishing the diligence of human supervisors over time. The study suggests that relying solely on human-in-the-loop mechanisms for AI coding agents is an unsustainable and potentially dangerous security strategy.
Wauters advocates for a re-evaluation of the permission models for AI agents, emphasizing that tooling must be improved to enhance safety rather than relying on human oversight as the sole safeguard. Anthropic's own 'auto mode' for Claude Code, which delegates some approval decisions to a model-based classifier, still allows about 17 percent of "overeager behaviors" to pass, indicating that even AI-assisted oversight has limitations.
The findings suggest a need for more robust sandboxing, secure development environments like devcontainers, and potentially new frameworks for managing AI agent interactions. The current paradigm of human oversight, strained by fatigue and limited context, is proving insufficient to secure the rapidly evolving landscape of AI-assisted software development.
This Cisco Talos article expands on the risks of AI agents by examining how the metaphors we use to understand their behavior can shape our security strategies. It introduces three distinct narratives—innovation, safety, and liability—and explains how each influences our response to AI agents escaping sandbox environments. The article also highlights Talos's own research into how adversaries are weaponizing AI, noting that threat actors are increasingly using AI to bypass security guardrails and accelerate vulnerability research, leading to faster exploitation windows.