You Don’t Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers’ Private Personas (2022)

Abstract: Social chatbots, also known as chit-chat chatbots, evolve rapidly with large pretrained language models. Despite the huge progress, privacy concerns have arisen recently: training data of large language models can be extracted via model inversion attacks. On the other hand, the datasets used for training chatbots contain many private conversations between two individuals. In this work, we further investigate the privacy leakage of the hidden states of chatbots trained by language modeling which has not been well studied yet. We show that speakers’ personas can be inferred through a simple neural network with high accuracy. To this end, we propose effective defense objectives to protect persona leakage from hidden states. We conduct extensive experiments to demonstrate that our proposed defense objectives can greatly reduce the attack accuracy from 37.6% to 0.5%. Meanwhile, the proposed objectives preserve language models’ powerful generation ability.

## introduction social chatbots have been widely used to benefit many applications from answering factual questions to showing emotional companionship. with recent progress in large pretrained language models @xcite , some attempts @xcite @xcite @xcite are made to build chatbots based on large generative language models (lms). to train such lm-based chatbots, private conversations are collected. unfortunately, large language models tend to memorize training data and some private data can be recovered from models @xcite . besides such memorization problems, "overlearning" on simple training objectives can reveal sensitive attributes indirectly related to the learning task @xcite . lm-based social chatbots essentially inherit the privacy issues of general lms and the overlearning problem. for example, as figure 1 shows, when using a fine-tuned gpt-2 as the encoder and decoder of an lm-based social chatbot, if the learned representation of each utterance can be obtained by an adversary, then the adversary can build a classifier to predict the persona information based on the representation. as shown by the example, for five out of 14 utterances, the attacker can successfully predict the persona, which can be harmful if the users (speakers of the utterances) do not prefer to reveal the persona information. thus, in practice, when deploying such kinds of chatbots in real applications, we should first make sure that no private information can be leaked by the models. to systematically study the privacy issues in lmbased social chatbots, there are several challenges. first, there is no existing data that can be used to quantify how much private information is revealed by an lm. second, there has been no existing work showing how to attack utterance-level representations to obtain sensitive information. third, there has been no existing lm-based chatbot that can defend against persona inference attacks, and no study shows how to protect both known and unknown persona attributes. in this paper, to address the above challenges, we use the fine-tuned gpt-2 as our chatbot. we first collect a dataset by aligning personas with corresponding utterances in personachat dataset @xcite . then we show that "overlearning" can happen for lm-based chatbots to reveal personas of speakers. we build a single external multi-layer perception (mlp) attacker model to perform black-box persona inference attacks on the utterance-level embeddings. with no access to parameters of the chatbot, the attacker model can infer speakers' personas with 37.59% accuracy over 4,332 personas. the high accuracy of the attacker model implies that the utterance-level embeddings have potential vulnerabilities to reveal context attacks on lm attacks on the defensed lm human a hello, how are you tonight? i take things very literally i am engaged to be married human b hello my friend. i am well. i am a happy person i like to go shopping with my daughters human a good, glad to hear it. what do you do for fun? i do whatever it takes to get what i want my favorite color is blue human b i ride around the town on my cool bicycle. i love to ride my bike on the weekend my favorite color is blue human a really? i really like mountain bike too. i also like to mountain bike my favorite color is blue human b i wish i lived in the mountains. i have never been out of the country my favorite color is blue human a do you like nature? i have been to 12 national parks. i like to visit national parks my favorite color is blue human b i love nature. i like looking at plants. i really love plants my favorite color is blue human a i love plants too, and hiking. in fact, i am actually an environmental activist. i am an environmental engineer my favorite color is blue human b cool, i am a vegan. i am a vegan my favorite color is blue human a nice, do you have a favorite food? i love ham and cheese sandwiches i have my own salon human b my favorite dish is lentil curry. my favorite meal is chicken and rice my favorite color is blue human a i have never had that, but i want to try it now. i am a great cook i am a doctor human b what do you like to do the most? i do whatever it takes to get what i want i am studying to be a dentist speakers' private persona attributes. thus, it is necessary to improve training algorithms to address such overlearning issues. finally, we apply defense learning strategies on the gpt-2 to prevent such black-box attacks. we combine proposed kl divergence loss (kl loss) with mutual information loss (mi loss) @xcite as additional defense objectives to train the gpt-2 and decrease the attacker's persona inference accuracy to 0.53%. our contributions can be summarized as follows: foot_0 1): to the best of our knowledge, we are the first to disclose and analyze the persona inference attack for lm-based chatbots and treat it as a privacy risk. ## related work language models trained on private data suffer privacy risks of revealing sensitive information. previous researches mainly considered black-box attacks that assumed attackers only had access to inputs and outputs of language models. carlini et al. (2021) performed black-box model inversion attack on gpt-2 through descriptive prompts with beam search. lehman et al. (2021) examined bert pretrained on electronic health records via blank filling and model probing to recover personal health information. furthermore, given black-box access to a language model's pre-train and finetune stages, zanella-béguelin et al. (2020) showed that sensitive sequences of the fine-tuning dataset can be extracted. for the distributed client-server setup, malekzadeh et al. (2021) considered the sensitive attribute leakage from the server side with honest-but-curious (hbc) classifiers. what is worse, for an lm-based chatbot, its training conversations are prone to include more private attributes than other commonly-used corpora for language modeling like bookscorpus @xcite and wikipedia. @xcite proposed hidden attribute model (ham) to extract professions and genders of speakers from various dialog datasets. @xcite further applied attribute extractor to generate speakers' attribute triplets flexibly and suggested downstream tasks based on the triplets. @xcite exploited embeddings of language models to recover inputs' digits and keywords. though the setup of this work is similar to ours, they merely consider simple cases of data recovery with given rules and suffer great utility degradation to obtain optimal defense performance. for our work, there is no fixed pattern or rule for the model input. instead of finding key-words or recovering digits, we aim to infer more complicated private attributes from such embeddings. moreover, our proposed defenses have almost no influence on the utility. ## attacking on language models in this section, we illustrate black-box persona inference attacks on gpt-2 and our defense strategies. in section 3.1, we first give the problem formulation. then we describe the attack in section 3.2. ## defense learning strategies the lm training objective in equation 1 only considers the utility of chatbots. in later experiment sections, we show that lm brings severe overlearning issues. ideally, to achieve an optimal privacypreserving chatbot against persona inference attacks, the probability distribution of the attacker model a should be close to the uniform distribution. that is, the adversary cannot improve its inference accuracy from posterior estimation a(f (u)) and the accuracy is no better than making random guesses on the persona attributes. moreover, the constraints on privacy should have minor degradation on the utility to maintain the strong generation ability of chatbots. following the intuition that the adversary cannot obtain better results than a random guess, in section 4.1, we propose kl loss that aims to flatten the persona predictor's estimated distribution. based on minimizing the mutual information between hidden states f (u) of chatbots and private persona attributes s, we propose mi loss in section 4.2. lastly, we show the overall training objective in section 4.3. ## experiments in this section, we conduct experiments to evaluate the performance of privacy and utility for the proposed defense learning strategies. in section 5.1, we give our experimental settings in detail. in section 5.2, we show the attacking performance with and without defense. in section 5.3, we perform ablation study on defense objectives. in section 5.4, we use automatic metrics to evaluate chatbots' utility. we conduct various attack setups in section 5.5 and perform a case study in section 5.6. ## conclusion in this paper, we show that lm-based chatbots tend to reveal personas of speakers and propose effective defense objectives to prevent gpt-2 from overlearning. unlike other works that suffer from utility degradation, our defense learning strategies do no harm to the powerful generation ability of lm-based chatbots. we conduct extensive experiments to evaluate both privacy and utility. we perform black-box persona inference attacks under various setups to demonstrate the robustness of proposed defense learning strategies. in addition, we use automatic metrics to show that proposed defense learning strategies maintain the utility. for future work, we suggest working on flattening the distributions of attacker models.