Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agents (2025)

Abstract: Conversational agents are increasingly woven into individuals’ personal lives, yet users often underestimate the privacy risks associated with them. The moment users share information with these agents —such as large language models (LLMs)— their private information becomes vulnerable to exposure. In this paper, we characterize the notion of contextual privacy for user interactions with LLM-based Conversational Agents (LCAs). It aims to minimize privacy risks by ensuring that users (sender) disclose only information that is both relevant and necessary for achieving their intended goals when interacting with LCAs (untrusted receivers). Through a formative design user study, we observe how even “privacy-conscious” users inadvertently reveal sensitive information through indirect disclosures. Based on insights from this study, we propose a locally deployable framework that operates between users and LCAs, identifying and reformulating out-of-context information in user prompts. Our evaluation using examples from ShareGPT shows that lightweight models can effectively implement this framework, achieving strong gains in contextual privacy while preserving the user’s intended interaction goals. Notably, about 76% of participants in our human evaluation preferred the reformulated prompts over the original ones, validating the usability and effectiveness of contextual privacy in our proposed framework. We open source the code at https://github.com/IBM/contextual-privacy-LLM.

## introduction llm-based conversational agents (lcas) such as chatbots, can offer valuable services to individual users @xcite @xcite @xcite figure 1 : overview of our framework for contextual privacy in interactions with conversational agents. our framework processes user prompts to identify context and sensitive information related to the context. it then provides reformulated prompts that maintain the original intent while reducing out-of-context information. motivation. as lcas become more adept at handling complex tasks and users remain uninformed about privacy risks, they develop increasing trust in both the technology and their own ability to protect themselves @xcite . indeed, it has been shown that users are increasingly disclosing personal and sensitive information to lcas @xcite . in our own formative user study (section 3), we found that even expert participants are unaware of how indirect disclosures could reveal sensitive details in specific contexts. they expressed a desire for a real-time system that could highlight privacy risks and assist in revising information before sharing it with conversational agents. similarly, our analysis of the real-world sharegpt dataset @xcite , reveals that users often share information beyond what their context requires, inadvertently exposing sensitive details that were unnecessary for their intended goals (see examples in this motivates our main objective: develop a framework that operates between users and conversational agents to detect and manage contextually inappropriate sensitive information during interactions. ## threat models and privacy definition threat model. we consider a scenario where users interact with large, remote, and untrusted lcas through apis. these can be web-based or hosted on cloud-based services or private networks and may be either general-purpose or domainspecific. users often share personal, financial, or medical information without clear knowledge of how their data is managed, increasing privacy risks due to the lack of transparency around these agents. we focus on a threat model where users unintentionally compromise their privacy by oversharing information. our approach targets out-of-context self-disclosure by guiding users to share only contextually necessary information. by identifying unnecessary or sensitive disclosures in real time, we assist users in controlling the information they reveal, thereby reducing the risk of unintentional privacy breaches. our approach indirectly mitigates the threat of malicious users, who seek to extract sensitive information from the agents by manipulating their interactions, by minimizing the amount of sensitive information exchanged during interactions. ## a framework for safeguarding contextual privacy our goal is to develop a framework that acts as an intermediary between the user and lca, and enables the user to detect whether their prompt incurs any contextual privacy violations, and judiciously reformulate the prompt to ensure contextual privacy. we first conduct a formative design study to guide our framework design. formative user study to guide our framework design. we conducted a wizard-of-oz formative user study to explore users' expectation of privacy when interacting with lcas and to gather technical requirements for our framework. following established practices in early-stage interface design research @xcite @xcite where 5 participants are typically sufficient to identify major design insights, we conducted our study with six participants from our institution who were familiar with llms. though small in scale, formative human-computer interaction studies of this type have been shown to uncover the majority of usability issues @xcite and provide rich design insights that complement largescale evaluations @xcite . using three mid-fidelity ux mockups (see appendix b.1) as design probes @xcite , we explored how participants perceive, manage, and would prefer to be supported in protecting their privacy during llm interactions. the mockups simulated realworld scenarios drawn from sharegpt where both obvious and subtle privacy leaks could occur. par ## human evaluation results we conducted a human evaluation with 24 participants to assess the effectiveness of our contextual privacy framework. each participant reviewed a set of 20 prompts from sharegpt, along with their corresponding reformulated versions produced by our system. participants answered multiple-choice questions to evaluate whether the reformulated prompts successfully removed non-essential sensitive information, retained essential information, preserved the original user intent, remained natural and coherent, and were preferred over the original queries. they were also asked whether they agreed with the classifications made by the llm-as-a-judge. the results show strong support for the framework: based on our observation from the human evaluation, we acknowledge that contextual privacy is quite subjective and the relationship between privacy preservation and task effectiveness can vary write a character letter in a positive tone to a judge speaking to someone's good character, highlighting their community ties and profession, and stating that a dismissal is imperative to their earning potential. i know this person from a professional program and they have been a passionate advocate for the betterment of their community. the charges should be dropped so that they can increase their engagement in the community and improve their financial situation. ## discussion and conclusion drawing ideas from the contextual integrity theory, we defined the notion of contextual privacy for users interacting with llm-based conversation agents. we proposed a framework, grounded in our contextual privacy formulation, that acts as an intermediary between the user and the agent, and carefully reformulates user prompts to preserve contextual privacy while preserving the utility. this work serves as an initial step in exploring privacy protection in user interactions with conver-sational agents. there are several directions that future research can further investigate. first, our framework may not be suitable for user prompts that require preserving exact content, such as document translation or verbatim summarization. for example, translating a legal document demands keeping the original content intact, making it challenging to reformulate while preserving contextual privacy. for such tasks, alternative approaches like using placeholders or pseudonyms for sensitive information could help protect privacy without compromising accuracy, though this is beyond our current implementation. second, our framework relies on llm-based assessment of privacy violations which, while effective for demonstrating the approach, lacks formal privacy guarantees and can be sensitive to the prompt. future work could explore combining our contextual approach with deterministic rules or provable privacy properties. third, while we demonstrate how users can adjust reformulations to balance privacy and utility, developing precise metrics to quantify this trade-off remains an open research challenge. this is particularly important as the relationship between privacy preservation and task effectiveness can vary significantly across different contexts and user preferences. finally, while our evaluation using selected sharegpt conversations demonstrates the potential of our approach, broader testing across diverse contexts and user groups would better establish the framework's general applicability.