P olicy QA : A Reading Comprehension Dataset for Privacy Policies (2020)

Abstract: Privacy policy documents are long and verbose. A question answering (QA) system can assist users in finding the information that is relevant and important to them. Prior studies in this domain frame the QA task as retrieving the most relevant text segment or a list of sentences from the policy document given a question. On the contrary, we argue that providing users with a short text span from policy documents reduces the burden of searching the target information from a lengthy text segment. In this paper, we present PolicyQA, a dataset that contains 25,017 reading comprehension style examples curated from an existing corpus of 115 website privacy policies. PolicyQA provides 714 human-annotated questions written for a wide range of privacy practices. We evaluate two existing neural QA models and perform rigorous analysis to reveal the advantages and challenges offered by PolicyQA.

## introduction security and privacy policy documents describe how an entity collects, maintains, uses, and shares users' information. users need to read the privacy policies of the websites they visit or the mobile applications they use and know about their privacy practices that are pertinent to them. however, prior works suggested that people do not read privacy policies because they are long and complicated @xcite , and confusing @xcite . hence, giving users access to a question answering system to search for answers from long and verbose policy documents can help them better understand their rights. in recent years, we have witnessed noteworthy progress in developing question answering (qa) systems with a colossal effort to benchmark highquality, large-scale datasets for a few application an essential characteristic of policy documents is that they are well structured as they are written by following guidelines set by the policymakers. besides, due to the homogeneous nature of different entities (e.g., amazon, ebay), their privacy policies have a similar structure. therefore, we can exploit the document structure (meta data) to form examples from existing corpora. in this paper, we present policyqa, a reading comprehension style question answering dataset with 25,017 question- policyqa (this work) privacyqa source website privacy policies mobile application privacy policies # policies 115 35 # questions 714 1,750 # annotations 25,017 3,500 question annotator domain experts mechanical turkers form of qa reading comprehension sentence selection answer type a sequence of words a list of sentences in contrast to privacyqa @xcite that focuses on extracting long text spans from policy documents, we argue that highlighting a shorter text span in the document facilitates the users to zoom into the policy and identify the target information quickly. to enable qa models to provide such short answers, policyqa provides examples with an average answer length of 13.5 words (in comparison, the privacyqa benchmark has examples with an average answer length of 139.6 words). we present a comparison between privacyqa and policyqa in in this work, we present two strong neural baseline models trained on policyqa and perform a thorough analysis to shed light on the advantages and challenges offered by the proposed dataset. the data and the implemented baseline models are made publicly available. foot_0 ## dataset policyqa consists question-passage-answer triples, curated from opp-115 @xcite . opp-115 is a corpus of 115 website privacy policies (3,792 segments), manually annotated by skilled annotators following the annotation schemes predefined by domain experts. the annotation schemes are composed of 10 data practice categories (e.g., first party collection/use, third party sharing/collection, user choice/control etc.). the data practices are further categorized into a set of practice attributes (e.g., personal information type, purpose, user type etc.). each practice attribute is associated with a predefined set of values. in "practice": first party collection/use "attribute": purpose "value": "additional service/feature" "startindexinsegment": 360 "endindexinsegment": 387 "selectedtext": "responding to your requests" "practice": third party sharing/collection "attribute": third party entity "value": "unnamed third party" "startindexinsegment": 573 "endindexinsegment": 596 "selectedtext": "third-party advertisers" the appendix (in in total, opp-115 contains 23,000 data practices, 128,000 practice attributes, and 103,000 annotated text spans. each text span belongs to a policy segment, and opp-115 provides its character-level start and end indices. we provide an example in question annotations. two skilled annotators manually annotate the questions. during annotation, the annotators are provided with the triple {practice, attribute, value}, and the associated text span. for example, given the triple {first party collection/use, personal information type, con-tact} and the associated text span "name, address, telephone number, email address", the annotators created questions, such as, ( 1 ) what type of contact information does the company collect?, (2) will you use my contact information?, etc. privacy practice proportion example question from policyqa first party collection/use 44.4 % why do you collect my data? third party sharing/collection 34.1 % do they share my information with others? data security 2.2 % do you use encryption to secure my data? data retention 1.7 % how long they will keep my data? user access, edit and deletion 3.1 % will you let me access and edit my data? user choice/control 11.0 % what use of information does the user choice apply to? policy change 1.9 % how does the website notify about policy changes? international and specific audiences 1.5 % what is the company's policy towards children? do not track 0.1 % do they honor the user's do not track preference? for a specific triple, the process is repeated for 5-10 randomly chosen samples to form a list of questions. we randomly assign a question from this list to the examples associated with the triple that were not chosen during the sampling process. in total, we considered 258 unique triples and created 714 individual questions. in we split opp-115 into 75/20/20 policies to form training, validation, and test examples, respectively. ## experiment in this section, we evaluate two neural question answering (qa) models on policyqa and present the findings from our analysis. baselines. policyqa frames the qa task as predicting the answer span that exists in the given policy segment. hence, we consider two existing neural approaches from literature as baselines for policyqa. the first model is bidaf @xcite that uses a bi-directional attention flow mechanism to extract the evidence spans. the second baseline is based on bert @xcite with two linear classifiers to predict the boundary of the evidence, as suggested in the original work. implementation. policyqa has a similar setting as squad @xcite . therefore, we pre-train the qa models using their default settings on the squad dataset. besides, we consider leveraging unlabeled privacy policies in fine-tuning the models, as noted below. similarly, to adapt bert to the privacy domain, we first fine-tune bert using masked language modeling @xcite based on the privacy policies and then train on policyqa. we adopt the default model architecture and optimization setup for the baseline methods. we detail the hyper-parameters in appendix (in ## related work the usable privacy project @xcite has made several attempts to automate the analysis of privacy policies @xcite . noteworthy works include identification of policy segments commenting on specific data practices @xcite , extraction of opt-out choices, and their provisions in policy text @xcite , and others @xcite . @xcite used a keyword-based technique to compare online privacy policies. natural language processing (nlp) techniques such as text alignment @xcite ), text classification @xcite @xcite and question answering @xcite @xcite has been studied in prior works to facilitate privacy policy analysis. among the question answering (qa) methods, @xcite framed the task as retrieving the most relevant policy segments as an answer, while @xcite presented a dataset and models to answer questions with a list of sentences. in comparison to the prior qa approaches, we encourage developing qa systems capable of providing precise answers by using policyqa. ## conclusion this work proposes policyqa, a reading comprehension style question answering (qa) dataset. pol-icyqa can contribute to the development of qa systems in the security and privacy domain that have a sizeable real-word impact. we evaluate two strong neural baseline methods on policyqa and provide thorough ablation analysis to reveal important considerations that affect answer span prediction. in our future work, we want to explore how transfer learning can benefit question answering in the security and privacy domain.