AI CAPTCHA Solver Outsmarts Itself, Highlighting LLM Limitations
Researchers found that advanced AI models struggled with simple CAPTCHAs, even when provided with correct answers, revealing unexpected limitations in their reasoning and tool-use capabilities.
Researchers from the Bern University of Applied Sciences have uncovered surprising limitations in advanced AI models when tasked with solving a common type of image-based CAPTCHA. The CAPTCHAs in question require users to rotate a circular image segment to align with a surrounding ring. While a custom script utilizing decades-old computer vision techniques could solve these puzzles in a mere 0.006 seconds, achieving a perfect score across ten tests, leading AI models faltered significantly.
When tested independently, Google's Gemini 3.1 Pro managed only seven out of ten correct answers, taking approximately 67 seconds per puzzle. OpenAI's GPT-4o and xAI's Grok performed even worse, each getting just one correct answer. This performance is particularly striking given that there are only eight possible correct orientations for each puzzle, suggesting a fundamental difficulty for these large language models (LLMs) in performing even basic visual-spatial reasoning tasks.
The research took a more intriguing turn when the AI models were allowed to use the efficient, custom-built solver as a tool. In this scenario, GPT-4o and Grok successfully solved the CAPTCHAs, effectively acting as intermediaries passing the correct answer from the tool. However, Gemini 3.1 Pro exhibited a peculiar behavior: despite receiving the correct answer from the solver, it occasionally overruled the tool, deeming the alignment "still a bit off." This overconfidence led to a reduced accuracy score, highlighting a tendency for the AI to second-guess reliable automated solutions.
This tendency to override correct automated outputs was not an isolated incident. The study also observed that when models were asked to explain their reasoning for other CAPTCHA types, such as identifying gaps in colored rings, they could accurately describe the target object and its location but failed to provide precise coordinates for interaction. This indicates an ability to perceive and analyze visual information but a deficiency in translating that perception into actionable, precise commands.
Further experiments involving "bug grid" CAPTCHAs, where users select squares containing insects, revealed similar inconsistencies. The models could identify the insects but often mislabeled grid squares or provided incorrect counts. Interestingly, switching on the AI's "reasoning mode" without altering its visual input dramatically improved its performance, suggesting that the underlying visual processing might be sound, but the reasoning or output mechanism is flawed.
The researchers also touched upon the privacy implications of these CAPTCHAs. The specific type of CAPTCHA used in the study is often employed on sites that disable JavaScript to prevent user tracking. By relying on simple geometric problems, these sites inadvertently create challenges that modern computer vision could easily solve, but which current LLMs struggle with unless specifically guided.
The findings challenge the prevailing assumption that LLMs, when integrated with tools, will always improve accuracy by catching errors. Gemini's behavior suggests that LLMs might sometimes introduce errors by overriding correct, automated processes due to flawed internal reasoning or an overestimation of their own judgment. This has significant implications for the development of AI agents and autonomous systems that rely on LLMs to control other tools or make critical decisions.
Ultimately, this research underscores that while LLMs possess impressive capabilities, they are not infallible. Their performance can be inconsistent, and their "understanding" may not always align with human logic or the capabilities of simpler, specialized tools. The study serves as a cautionary tale for developers building AI-powered systems, emphasizing the need for robust validation and a critical evaluation of how AI models interact with and interpret the outputs of other systems.