OpenAI Compaction Flaw Exposed Hidden AI Reasoning, Researchers Find
A flaw in OpenAI's conversation compaction feature allowed researchers to extract a model's hidden reasoning, bypassing intended protections.
Security researchers at Xint have uncovered a significant vulnerability in OpenAI's AI models, demonstrating how a feature designed to optimize conversation storage could be exploited to reveal the AI's internal reasoning processes. The flaw, identified in OpenAI's conversation compaction mechanism, allowed researchers to bypass security measures intended to keep the AI's step-by-step thought processes private.
OpenAI's reasoning models typically work through a problem before providing an answer. The service is designed to return only the final output to users, with the underlying reasoning kept encrypted and inaccessible. This encrypted reasoning is crucial for preventing 'distillation,' a process where attackers train cheaper models on the proprietary outputs of more advanced ones. However, the newly discovered compaction flaw effectively circumvented these protections.
Xint researchers successfully tested the technique against OpenAI's gpt-5.6-sol model using custom math and algorithm puzzles. By manipulating the compaction process, they were able to extract what appeared to be the model's reasoning in plain text. While Xint could not definitively prove the extracted text was an exact, word-for-word replica of the model's original reasoning due to the lack of a direct comparison, subsequent tests on cheaper models that repeated the output suggested the compaction process itself did not introduce significant errors.
The compaction feature is intended to summarize long conversations into smaller, encrypted data points that retain context. Xint's exploit involved sending a task to the gpt-5.6-sol model, capturing its encrypted reasoning, and then using the compaction function on the same model. The researchers instructed the compaction process to include the reasoning verbatim, with specific formatting changes (spaces to tabs) to bypass OpenAI's content filters.
This method proved effective against a security fix OpenAI had implemented in August. Previously, researchers had demonstrated that encrypted reasoning could be replayed into weaker models, causing them to output the reasoning in plain text. OpenAI had patched this 'Stolen Thoughts' vulnerability, but Xint's compaction exploit bypassed this fix because the compaction step occurred on the same model that generated the reasoning, thus avoiding the cross-model replay detection.
The technique builds upon Xint's prior research, known as GhostCompact, which showed that the compaction process could be manipulated to execute arbitrary tasks. Xint has made the code for this latest discovery publicly available on their GitHub repository. OpenAI has since confirmed the vulnerability and applied a fix on October 23, reportedly rewarding Xint with a $6,000 bounty.
While Xint's tests focused on gpt-5.6-sol, it remains unclear if OpenAI's newer GPT-6 Astra model is also affected. The researchers also noted that this specific exploit path might be one of many, suggesting that any server-side process that decrypts and handles model reasoning could potentially be a vector for leakage.
This incident highlights the ongoing challenges in securing advanced AI models, particularly concerning the protection of proprietary training data and internal model processes. OpenAI has previously disclosed efforts to disrupt campaigns, some linked to China-based actors, aimed at stealing AI reasoning, underscoring the strategic importance of these intellectual property assets in the competitive AI landscape.