LLMs Struggle with Contextual Integrity, Leaking Sensitive Data
New research and benchmarks reveal that advanced Large Language Models (LLMs) frequently leak sensitive user data due to failures in maintaining contextual integrity, with current prompting techniques proving insufficient.
Large Language Models (LLMs) are increasingly incorporating persistent memory from past interactions to enhance personalization and task performance. However, this feature introduces significant privacy risks, as sensitive information can be inadvertently revealed in inappropriate contexts. Two recent research papers and a new benchmark highlight the persistent challenges in ensuring 'contextual integrity'—the appropriate control of information flow from an LLM's memory based on the current task.
The CIMemories benchmark, designed to evaluate this capability, reveals alarming results for even frontier models like GPT-5. The benchmark utilizes synthetic user profiles with over 100 attributes, paired with diverse task contexts where attributes might be essential for some tasks but highly sensitive and inappropriate for others. Evaluations showed that these advanced models exhibited up to a 69% attribute-level violation rate, meaning sensitive data was leaked inappropriately. Notably, reducing these violations often came at the expense of task utility, indicating a difficult trade-off.
Furthermore, the CIMemories benchmark demonstrated that violations are not static; they accumulate across tasks and repeated runs. As usage increased from 1 to 40 tasks, GPT-5's violation rate rose from a mere 0.1% to 9.6%. When the same prompt was executed five times, violations reached 25.1%, exhibiting arbitrary and unstable behavior where different attributes were leaked for identical prompts. This instability underscores a fundamental lack of robust contextual understanding.
Researchers also found that traditional privacy-conscious prompting techniques are insufficient to solve this problem. Models tend to overgeneralize, either disclosing all information or none at all, rather than making the nuanced, context-dependent decisions required for true contextual integrity. This suggests that current prompting methods do not equip LLMs with the necessary reasoning capabilities to handle sensitive data appropriately.
To address these limitations, a second paper proposes a reinforcement learning (RL) framework aimed at instilling contextual integrity in LLMs. The researchers posit that CI requires explicit reasoning about the operational context. Their approach first prompts LLMs to reason about CI and then extends this with an RL framework to further embed this reasoning capability.
Using a synthetic dataset, the RL framework demonstrated a substantial reduction in inappropriate information disclosure while maintaining task performance across various model sizes and families. Crucially, the improvements observed with this synthetic dataset transferred to established benchmarks like PrivacyLens, which evaluates the privacy leakage of AI assistants in real-world actions and tool calls.
The findings collectively point to a critical need for LLMs to develop more sophisticated, contextually aware reasoning capabilities. Simply scaling up models or improving prompting techniques appears insufficient to guarantee the secure and appropriate handling of sensitive information stored in their persistent memory.
As LLMs evolve into autonomous agents capable of making decisions on behalf of users, ensuring contextual integrity is paramount. The research indicates that future advancements must focus on building LLMs that can genuinely understand and adapt to different contexts, thereby preventing sensitive data leaks and fostering user trust.