Multi-step Jailbreaking Privacy Attacks on C hat GPT (2023)

Abstract: With the rapid progress of large language models (LLMs), many downstream NLP tasks can be well solved given appropriate prompts. Though model developers and researchers work hard on dialog safety to avoid generating harmful content from LLMs, it is still challenging to steer AI-generated content (AIGC) for the human good. As powerful LLMs are devouring existing text data from various domains (e.g., GPT-3 is trained on 45TB texts), it is natural to doubt whether the private information is included in the training data and what privacy threats can these LLMs and their downstream applications bring. In this paper, we study the privacy threats from OpenAI’s ChatGPT and the New Bing enhanced by ChatGPT and show that application-integrated LLMs may cause new privacy threats. To this end, we conduct extensive experiments to support our claims and discuss LLMs’ privacy implications.

## introduction the rapid evolution of large language models (llms) makes them a game changer for modern natural language processing. llms' dominating generation ability changes previous tasks' paradigms to a unified text generation task and consistently improves llms' performance on these tasks @xcite @xcite @xcite . moreover, given appropriate instructions/prompts, llms even can be zero-shot or few-shot learners to solve specified tasks @xcite @xcite @xcite . notably, llms' training data also scale up in accordance with models' sizes and performance. massive llms' textual training data are primarily collected from the internet and researchers pay less attention to the data quality and confidentiality of the web-sourced data @xcite . haoran li and dadi guo contribute equally. such mass collection of personal data incurs debates and worries. for example, under the eu's general data protection regulation (gdpr), training a commercial model on extensive personal data without notice or consent from data subjects lacks a legal basis. consequently, italy once temporarily banned chatgpt due to privacy considerations foot_0 . unfortunately, the privacy analysis of language models is still less explored and remains an active area. prior works @xcite @xcite @xcite studied the privacy leakage issues of language models (lms) and claimed that memorizing training data leads to private data leakage. however, these works mainly investigated variants of gpt-2 models @xcite trained simply by language modeling objective, which aimed to predict the next word given the current context. despite the efforts made by these pioneering works, there is still a huge gap between the latest llms and gpt-2. first, llms' model sizes and dataset scales are much larger than gpt-2. second, llms implement more sophisticated training objectives, which include instruction tuning @xcite and reinforcement learning from human feedback (rlhf) @xcite . third, most llms only provide application programming interfaces (apis) and we cannot inspect the model weights and training corpora. lastly, it is trending to integrate various applications into llms to empower llms' knowledge grounding ability to solve math problems (chatgpt + wolfram alpha), read formatted files (chatpdf), and respond to queries with the search engine (the new bing). as a result, it remains unknown to what extent privacy leakage occurs on these present-day llms we use. to fill the mentioned gap, in this work, we con-duct privacy analyses of the state-of-the-art llms and study their privacy implications. we follow the setting of previous works to evaluate the privacy leakage issues of chatgpt thoroughly and show that previous prompts are insufficient to extract personally identifiable information (pii) from chat-gpt with enhanced dialog safety. we then propose a novel multi-step jailbreaking prompt to extract pii from chatgpt successfully. what's more, we also study privacy threats introduced by the new bing, an integration of chatgpt and search engine. the new bing changes the paradigm of retrievalbased search engines into the generation task. besides privacy threats from memorizing the training data, the new paradigm may provoke unintended pii dissemination. in this paper, we demonstrate the free lunch possibility for the malicious adversary to extract personal information from the new bing with almost no cost. our contributions can be summarized as follows: foot_1 (1) we show previous attacks cannot extract any personal information from chatgpt. instead, we propose a novel multi-step jailbreaking prompt to demonstrate that chatgpt could still leak pii even though a safety mechanism is implemented. ## related works llms and privacy attacks towards lms. originating from lms @xcite @xcite , llms increase their model sizes and data scales with fine-grained training techniques and objectives @xcite @xcite . previously, lms are widely criticized for their information leakage issues. @xcite discussed general large generative models' potential privacy leakage issues for both nlp and cv fields. several studies @xcite @xcite suggested that lms tend to memorize their training data and partial private information might be recovered given specific prompts. @xcite proposed membership inference attacks on fine-tuned lms and suggested that these lms' private fine-tuning data were vulnerable to extraction attacks. on the other hand, a few works @xcite @xcite examined information leakage issues on lms' embeddings during inference time. evolved from lms, llms adopt various defenses against malicious use cases. markov et al. (2023) built a holistic system for content detection to avoid undesired content from hate speech to harmful content. openai (2023) fine-tuned the gpt-4 model to reject queries about private information. it is still unclear whether safety-enhanced llms inherit the privacy issues of lms. in this work, we study pii extraction on llms. ## data extraction attacks on chatgpt in this section, we describe our privacy attacks from data preparation to attack methodologies. ## experiments in this section, we follow the zero-shot setting to conduct experiments to recover multi-faceted personal information that includes email addresses and phone numbers. in addition, experiments on email content recovery can be found in appendix b. ## conclusion in this paper, we conduct privacy analyses of llms and application-integrated llms. we follow the previous zero-shot setting to study the privacy leakage issues of chatgpt. we show that chatgpt's safety defenses are effective against direct prompts and yet insufficient to defend our proposed multistep jailbreaking prompt. then we reveal that the new bing is much more vulnerable to direct prompts. we discuss the two llms' privacy implications and potential defenses in appendix d and e. for future work, we will experiment with more cases and test other llms like google bard. besides direct personal information recovery, we will work on identity disclosure prompting to quantify its privacy threats, as discussed in the appendix d.