Personal Information Leakage Detection in Conversations (2020)

Abstract: The global market size of conversational assistants (chatbots) is expected to grow to USD 9.4 billion by 2024, according to MarketsandMarkets. Despite the wide use of chatbots, leakage of personal information through chatbots poses serious privacy concerns for their users. In this work, we propose to protect personal information by warning users of detected suspicious sentences generated by conversational assistants. The detection task is formulated as an alignment optimization problem and a new dataset PERSONA-LEAKAGE is collected for evaluation. In this paper, we propose two novel constrained alignment models, which consistently outperform baseline methods on Moreover, we conduct analysis on the behavior of recently proposed personalized chit-chat dialogue systems. The empirical results show that those systems suffer more from personal information disclosure than the widely used Seq2Seq model and the language model. In those cases, a significant number of information leaking utterances can be detected by our models with high precision.

## introduction according to opus research 2 , 4.5 billion dollars will be invested in conversational assistants (chatbots) by 2021. among diverse types of chatbots, google duplex, first introduced at google i/o 2018, represents the kind of ai personal assistants (pas) that act on behalf of people to perform simple tasks, such as making reservations at restaurants and hair salons. in order to successfully complete those tasks, pas are granted the access to personal information (pi) of their owners, such as number of children, working hours, home address, and vacation plans. thus, these pas pose privacy concerns when they communicate with real-life people, or other bots in natural language. another major source of personal information leakage is online social networks, which store a huge amount of possibly sensitive information on users and their interactions @xcite . however, a recent study shows that none of the popular social network platforms (facebook, wechat, google+, etc) have developed a perfectly non-leaky privacy protection mechanism @xcite . in addition, internet users (including a vast number of children and teenagers) often show a phenomenon called privacy paradox, which states that even users with high level of privacy concerns do not always take appropriate actions although those measures are fairly easy to perform @xcite ). as an unfortunate example, children's privacy is often unconsciously compromised by their parents' online behaviour, such as online posting and mes-saging @xcite . an ideal privacy protection solution is not to stop using pas or discourage online socialization, but to have the ability to control the dissemination of personal information @xcite . personal information can be dispersed through various types of media. in this work, we focus on natural language utterances in conversations articulated by pas or humans. the ways of controlling such textual information vary significantly w.r.t. platforms, pas, user preferences, and social circles. since there is no universally applicable control strategy, we take the first step towards privacy protection by designing a personal information leakage detection module (pild) that warns users or alerts pas whenever an utterance is associated with personal information, as illustrated in figure 1 . the warning module gives authorized users the capability to control information leakage from the start. then, it is up to users and the design of pas to decide how they deal with utterances leaking personal information. pas will communicate with other interlocutors using secure or approved utterances. we formulate detection of utterances causing personal information leakage as a text alignment problem, which aims to link information leaking utterances to the corresponding textual descriptions of personal information. we consider personal information provided in text, because i) user profiles on popular social network platforms include a significant proportion of textual descriptions, and ii) it is natural for users to share their information with pas in natural language. figure 2 demonstrates an example of aligning utterances in a dialogue with a set of personal information descriptions. those red lines depict the ground-truth alignments between utterances and personal information descriptions. the true alignments are sparse as not all utterances leak personal information, e.g., u1, u3 and u6. meanwhile, an utterance may be associated with more than one descriptions of personal information, e.g., u2 and u4, and vice versa. in the absence of direct supervision signals, we explore low annotation-cost solutions to this text alignment problem by considering a weakly supervised setting. in this setting, we only know who speaks what and what are the pi descriptions of each interlocutor during training, without knowing true alignments. the additional challenges are imposed by the complex relationships between utterances and descriptions of pi, which could be sparse alignment, and one-to-one, one-to-many, many-toone, or many-to-many mapping. to address the aforementioned challenges, we propose two models sharp-max and sparse-max by formulating the text alignment problem as constrained optimization problems. the training procedure takes the form of contrastive learning @xcite . herein, we encourage aligning an utterance with the descriptions of its interlocutor subject to sparsity constraints, while penalizing its alignments with those of other speakers. thus, sentence-level alignments are not employed during training. the main contributions are the following: ## alignment models in this section, we formally define the problem of pi leakage detection as text alignment between utterances and descriptions of pi in the weakly supervised setting, followed by presenting the architecture shared by the two proposed alignment models sparse-max and sharp-max. the two models differ in the sparsity regularization for alignments during training. we then detail the training algorithms as well as how to derive the regularizers. ## empirical results and analysis in this section, we analyze our experimental results. our experiments are designed to answer the following research questions (rqs), ## related work recently, privacy and fairness started to attract more and more attention from nlp community. sensitive information was removed from latent representations via adversarial training @xcite and differential privacy @xcite , achieving fair decisions. privacy-aware text rewriting methods suggested to generate new sentences with less sensitive information @xcite @xcite . our work serves as a component that detects the sentences requires rewriting. another line of research aims to identify mentions of pre-defined semantic categories indicating sensitive information in text (microsoft; @xcite , such as bank account and phone number. in our setting, sensitive information can be expressed in any syntactic structures, including events conveyed in whole sentences, such as "i have got less than 5 hours of sleep each night for years" is associated with the persona "i have sleep disorders for many years.". the setting of our work is more general, as we focus on open-domain personal information written in natural language, which is not limited to mentions of fixed semantic categories in sentences or sensitivity labels of sentences. our work places an emphasis on privacy concerns in conversations @xcite @xcite @xcite . in recent research, several works have attempted to improve the engagement and diversity of chit-chat dialogue system @xcite @xcite and goal-oriented dialogue system @xcite . with the rapid development of personalized dialogue systems, pild module is expected to address the privacy concerns @xcite . @xcite improved the coherence and consistency of a dialogue using natural language inference (nli) @xcite . dialogue-nli dataset could be utilized to train retrieval models, however, it does not directly address the privacy concerns. in contrast, our dataset i) considers all possible leakage pairs, and ii) includes sensitivity annotations of all pi descriptions. our problem setting was inspired by an imagesentence alignment problem, given pairs of image sets and documents @xcite . similar problems were also explored in the context of align-ing image fragments with words @xcite . in this paper, we considered utterances and pi descriptions from the same interlocutor as coarse-grained alignment signals, which are in the same modality. ## conclusions and future work we formulate protection of personal information in conversations as a weakly supervised alignment between personal information and dialogue utterances. to tackle this task, we proposed two new alignment models and created a dataset persona-leakage for evaluation. our experimental results demonstrate the effectiveness of our methods in comparison with the competitive baselines on that dataset. further analysis on human-bot dialogue performance demonstrated the potential privacy risks with advanced personalized dialogue techniques. this work is the first step towards fully preventing leakage of privacy in text, which still requires pas or users to select and hide sensitive information. we hope this work and the dataset will pave the way for the research on privacy leakage in conversations. in the future, we will explore fullfledged solutions to address the privacy concerns of both humans and dialogue systems.