Intent Classification and Slot Filling for Privacy Policies (2021)

Abstract: Understanding privacy policies is crucial for users as it empowers them to learn about the information that matters to them. Sentences written in a privacy policy document explain privacy practices, and the constituent text spans convey further specific information about that practice. We refer to predicting the privacy practice explained in a sentence as intent classification and identifying the text spans sharing specific information as slot filling. In this work, we propose PolicyIE, an English corpus consisting of 5,250 intent and 11,788 slot annotations spanning 31 privacy policies of websites and mobile applications. PolicyIE corpus is a challenging real-world benchmark with limited labeled examples reflecting the cost of collecting large-scale annotations from domain experts. We present two alternative neural approaches as baselines, (1) intent classification and slot filling as a joint sequence tagging and (2) modeling them as a sequence-to-sequence (Seq2Seq) learning task. The experiment results show that both approaches perform comparably in intent classification, while the Seq2Seq method outperforms the sequence tagging approach in slot filling by a large margin. We perform a detailed error analysis to reveal the challenges of the proposed corpus.

## introduction privacy policies inform users about how a service provider collects, uses, and maintains the users' information. the service providers collect the users' data via their websites or mobile applications and analyze them for various purposes. the users' data often contain sensitive information; therefore, the users must know how their information will be used, maintained, and protected from unauthorized and unlawful use. privacy policies are meant to explain all these use cases in detail. this makes * equal contribution. listed by alphabetical order. privacy policies often very long, complicated, and confusing @xcite . as a result, users do not tend to read privacy policies @xcite @xcite , leading to undesirable consequences. for example, users might not be aware of their data being sold to third-party advertisers even if they have given their consent to the service providers to use their services in return. therefore, automating information extraction from verbose privacy policies can help users understand their rights and make informed decisions. in recent years, we have seen substantial efforts to utilize natural language processing (nlp) techniques to automate privacy policy analysis. in literature, information extraction from policy documents is formulated as text classification @xcite @xcite ), text alignment @xcite , and question answering (qa) @xcite @xcite . although these approaches effectively identify the sentences or segments in a policy document relevant to a privacy practice, they lack in extracting fine-grained structured information. as shown in the first example in the challenge in training models to extract finegrained information is the lack of labeled examples. annotating privacy policy documents is expensive as they can be thousands of words long and requires domain experts (e.g., law students). therefore, prior works annotate privacy policies at the @xcite data collector: first party entity may also @xcite action or display @xcite data provider: user @xcite data collected: user online activities/profiles and [icon or profile photo] data collected: user online activities/profiles on [marketing purpose or press releases] purpose: advertising/marketing . privacy practice. data collection/usage @xcite data sharer: first party entity do @xcite polarity: negation [sell] action @xcite data provider: user [personal information] data shared: general data to [third parties] data receiver: third party entity . privacy practice. data sharing/disclosure to facilitate fine-grained information extraction, we present policyie, an english corpus consisting of 5,250 intent and 11,788 slot annotations over 31 privacy policies of websites and mobile applications. we perform experiments using sequence tagging and sequence-to-sequence (seq2seq) learning models to jointly model intent classification and slot filling. the results show that both modeling approaches perform comparably in intent classification, while seq2seq models outperform the sequence tagging models in slot filling by a large margin. we conduct a thorough error analysis and categorize the errors into seven types. we observe that sequence tagging approaches miss more slots while seq2seq models predict more spurious slots. we further discuss the error cases by considering other factors to help guide future work. we release the code and data to facilitate research. 1 2 construction of policyie corpus ## model & setup policyie provides annotations of privacy practices and corresponding text spans in privacy policies. we refer to privacy practice prediction for a sentence as intent classification and identifying the text spans as slot filling. we present two alternative approaches; the first approach jointly models intent classification and slot tagging @xcite , and the second modeling approach casts the problem as a sequence-to-sequence learning task @xcite . ## experiment results & analysis we aim to address the following questions. 1. how do the two modeling approaches perform on our proposed dataset ( § 4.1)? 2. how do they perform on different intent and slot types ( § 4.2)? 3. what type of errors do the best performing models make ( § 4.3)? ## related work automated privacy policy analysis automating privacy policy analysis has drawn researchers' attention as it enables the users to know their rights and act accordingly. therefore, significant research efforts have been devoted to understanding privacy policies. earlier approaches @xcite designed rule-based pattern matching techniques to extract specific types of information. under the usable privacy project @xcite , several works have been done @xcite @xcite @xcite @xcite . no-table works leveraging nlp techniques include text alignment @xcite ), text classification @xcite @xcite , and question answering (qa) @xcite @xcite . @xcite is the most closest to our work that used named entity recognition (ner) modeling technique to extract third party entities mentioned in policy documents. our proposed policyie corpus is distinct from the previous privacy policies benchmarks: opp-115 @xcite uses a hierarchical annotation scheme to annotate text segments with a set of data practices and it has been used for multilabel classification @xcite and question answering @xcite ; privacyqa @xcite frame the qa task as identifying a list of relevant sentences from policy documents. recently, @xcite created a dataset by tagging documents from opp-115 for privacy practices and uses ner models to extract them. in contrast, policyie is developed by following semantic parsing benchmarks, and we model the task following the nlp literature. intent classification and slot filling voice assistants and chat-bots frame the task of natural language understanding via classifying intents and filling slots given user utterances. several benchmarks have been proposed in literature covering several domains, and languages @xcite @xcite @xcite @xcite . our proposed policyie corpus is a new addition to the literature within the security and privacy domain. policyie enables us to build conversational solutions that users can interact with and learn about privacy policies. ## conclusion this work aims to stimulate research on automating information extraction from privacy policies and reconcile it with users' understanding of their rights. we present policyie, an intent classification and slot filling benchmark on privacy policies with two alternative neural approaches as baselines. we perform a thorough error analysis to shed light on the limitations of the two baseline approaches. we hope this contribution would call for research efforts in the specialized privacy domain from both privacy and nlp communities.