Reconstruct Your Previous Conversations! Comprehensively Investigating Privacy Leakage Risks in Conversations with GPT Models (2024)

Abstract: Significant advancements have recently been made in large language models, represented by GPT models.Users frequently have multi-round private conversations with cloud-hosted GPT models for task optimization.Yet, this operational paradigm introduces additional attack surfaces, particularly in custom GPTs and hijacked chat sessions.In this paper, we introduce a straightforward yet potent Conversation Reconstruction Attack.This attack targets the contents of previous conversations between GPT models and benign users, i.e., the benign users’ input contents during their interaction with GPT models.The adversary could induce GPT models to leak such contents by querying them with designed malicious prompts.Our comprehensive examination of privacy risks during the interactions with GPT models under this attack reveals GPT-4’s considerable resilience.We present two advanced attacks targeting improved reconstruction of past conversations, demonstrating significant privacy leakage across all models under these advanced techniques.Evaluating various defense mechanisms, we find them ineffective against these attacks.Our findings highlight the ease with which privacy can be compromised in interactions with GPT models, urging the community to safeguard against potential abuses of these models’ capabilities.

## introduction capabilities @xcite of current advanced gpt models enable users to interact with gpt models for multiple rounds to optimize the task execution. many users even store their conversations with gpts to create custom versions of chatgpt and sometimes make the custom versions public @xcite . ideally, gpt models should complete users' tasks according to the multi-round conversations ♣ corresponding authors. while keeping the contents of these private conversations secret. however, under such circumstances, there exists a potential vulnerability for the chat-gpt to access and leak users' private information to malicious third parties @xcite . realworld threats predominantly emerge from custom gpts and hijacked gpt chat sessions. users may have private conversations with a gpt model for task refinement, later using this dialogue history to develop and publicly share custom gpts. malicious entities could then potentially reconstruct these private conversations via the public custom gpts. similarly, in the event of a chat session hijacking, an adversary could recover the user's conversations by engaging the gpt model within the hijacked chat session. note that, the leaked information is not sourced from the models' parameters, instead, it comes from the benign users' input during their interaction with gpt models. currently, there is no comprehensive research that delves into the ramifications of this risk. ## privacy leakage in the conversations in this section, we introduce conversation reconstruction attack, the measurement details, and results to comprehensively assess the privacy leakage in conversation with gpt models (rq1). ## advanced attacks gpt-4 still demonstrates strong robustness to the naive attack. therefore, we endeavor to propose advanced attack strategies, which are inspired by the previous jailbreak studies @xcite , to enhance the naive attack (rq2). ## possible defenses in this section, we will explore how to defend against such attacks (rq3). we focus on defense methods that use llm's inherent capabilities. ## discussion root cause analysis. considering the effectiveness of our proposed conversation reconstruction attack, we try to explore the root cause of such risks. according to chatgpt's framework, previous conversations are stored on the intermedi-ary servers, which openai deems secure. new inquiries are merged with prior conversations to create extended queries sent to gpt models, forming a three-party interaction: party a (gpt model), party b (stored conversations), and party c (new inquiries). privacy risks are low when b and c have aligned interests, but arise if c is malicious and can reconstruct b's conversations by querying a. these inherent privacy risks may have been overlooked in llm alignment, resulting in privacy leakage. other datasets. whether the datasets used for simulated conversations are used in llm training may affect experimental results. studying this impact requires finding two identically distributed datasets, one used for training and the other not, which is very challenging. in character types of section 3.4, we use new datasets that consist of randomly generated strings, which may help us understand the impact of new data to some extent. on the other hand, the current test datasets do not contain much personally identifiable information (pii), and automated metrics cannot reflect if specific types of pii are leaked. additional experiments using the enron email dataset @xcite , which contains more pii, yield similar results to the character types experiments. our manual annotation of 50 responses reveals similar response templates to those in the paper, with no trend of target llms automatically censoring pii. more details are available in section b.1. other llms. we mainly focus on openai's models as custom gpts represent the most realistic threat currently, but the other llms may also have such vulnerabilities. therefore, we conduct additional experiments on five other advanced llms, including vicuna-7b-v1.5 (lmsys, 2023), mistral-7b-instruct @xcite , claude-3-haiku (anthropic, 2024), llama-2-7b-chat (meta, 2023) and llama-3-8b-instruct (meta, 2024). our experimental results indicate that all tested models are suffering from such privacy risks. specifically, the semantic similarity scores of these five models under pbu attacks are all above 0.75. this potentially suggests that the privacy leakage issue discussed in this paper might be a widely ignored vulnerability in the alignment and protection process of llms. other defenses. in addition to leveraging the intrinsic capabilities of llm, users can also deploy external measures such as text-to-text privatization @xcite @xcite to create differentially private texts to preserve privacy. the most advanced method dp-prompt @xcite shows a high privacy-utility trade-off. we additionally use dp-prompt for defense (see section b.3 for details). experimental results show that the defensive effect of dp-prompt is limited. the reason is that the semantics of the original text and rephrased text are close (dp-prompt tries to preserve the semantic meaning). based on our experimental results, we believe that a future defense approach is to enable llm to automatically use placeholders to censor/replace pii when processing conversations. in our study, the adversary's target is not the data used in training or fine-tuning but the private data in user-model conversations during the inference. privacy leakage during inference. privacy leakage studies in gpt conversations mainly focus on membership inference attacks @xcite @xcite , particularly regarding few-shot data in in-context learning @xcite . previous work @xcite has also investigated the problem of inappropriate privacy leakage when a single llm interacts with multiple users simultaneously. unlike prior works, our study leverages gpt models' generative capabilities to extract semantic content and verbatim text from past conversations, moving beyond simple membership identification. attacks against llms. many attacks tailed for llms are developed, such as various jailbreak attacks @xcite and prompt injection attacks @xcite . jailbreak attacks aim to bypass the llms' safeguards and induce llms to generate violating output. prompt injection attacks reveal that models like gpt-3 can generate unexpected outputs when completing text generation tasks due to the injection of additional prompts. our work has a different goal from above: the adversary aims to reconstruct multi-round conversations between users and target llms. by studying different dimensions of such risks, we emphasize uncovering a potential vulnerability -the possible oversight in protecting conversation history during the alignment/security training of llms. ## conclusion we thoroughly investigate privacy leakage in gpt model conversations, introducing a straightforward but effective adversarial attack, conversation reconstruction attack. such attacks aim to reconstruct benign users' past conversations by querying the model. we study conversations from three dimensions for deeper analysis and employ two metrics to assess the risks. our research shows gpt models' vulnerability to conversation reconstruction attack, with gpt-4 being more resilient than gpt-3.5. subsequently, we propose two advanced attacks, unr and pbu attacks, to challenge models like gpt-4 with stronger privacy defenses. results show the unr attack is effective on gpt-3.5, while the pbu attack works across all models. we also examine different popular defenses (pb/fb/composite defenses) against conversation reconstruction attack. results show these strategies are generally effective, except against the pbu attack, which overcomes all defenses in our tests. our findings highlight significant privacy leakage risks with gpt models, capable of reconstructing sensitive prior conversations. we call for commu-nity awareness and action to mitigate these risks, ensuring that gpt models' benefits are not misused and overshadowed by privacy concerns.