PLUE : Language Understanding Evaluation Benchmark for Privacy Policies in E nglish (2023)

Abstract: Privacy policies provide individuals with information about their rights and how their personal information is handled. Natural language understanding (NLU) technologies can support individuals and practitioners to understand better privacy practices described in lengthy and complex documents. However, existing efforts that use NLU technologies are limited by processing the language in a way exclusive to a single task focusing on certain privacy practices. To this end, we introduce the Privacy Policy Language Understanding Evaluation (PLUE) benchmark, a multi-task benchmark for evaluating the privacy policy language understanding across various tasks. We also collect a large corpus of privacy policies to enable privacy policy domain-specific language model pre-training. We evaluate several generic pre-trained language models and continue pre-training them on the collected corpus. We demonstrate that domain-specific continual pre-training offers performance improvements across all tasks. The code and models are released at https://github.com/JFChi/PLUE .

## introduction privacy policies are documents that outline how a company or organization collects, uses, shares, and protects individuals' personal information. without a clear understanding of privacy policies, individuals may not know how their personal information is being used or who it is being shared with. the privacy violation might cause potential harm to them. however, privacy policies are lengthy and complex, prohibiting users from reading and understanding them in detail @xcite @xcite . various natural language understanding (nlu) technologies have recently been developed to understand privacy policies @xcite @xcite @xcite @xcite . these tasks focus on understanding specific privacy practices at different syntax or semantics levels and require significant effort for data annotations (e.g., domain experts). it is hard to develop generic pre-trained language models (e.g., bert @xcite ) with task-specific fine-tuning using limited annotated data. besides, the unique characteristics of privacy policies, such as reasoning over ambiguity and vagueness, modality, and document structure @xcite , make it challenging to directly apply generic pre-trained language models to the privacy policy domain. to address these problems and encourage research to develop nlu technologies in the privacy policy domain, we introduce the privacy policy language understanding evaluation (plue) benchmark, to evaluate the privacy policy language understanding across six tasks, including text classification, question answering, semantic parsing, and named-entity recognition. plue also includes a pre-training privacy policy corpus that we crawl from the websites to enable privacy policy domainspecific language model pre-training. we use this corpus to pre-train bert @xcite , roberta @xcite , electra @xcite , and spanbert @xcite and finetune them on the downstream tasks. we demonstrate that domain-specific continual pre-training offers performance improvements across all tasks. we will release the benchmark to assist natural language processing (nlp) researchers and practitioners in future exploration. ## policy language understanding evaluation (plue) benchmark plue is centered on six english privacy policy language understanding tasks. the datasets and tasks are selected based on the following principles: (1) usefulness: the selected tasks can help practitioners based on the role of the slots in privacy practices, policyie groups them into type-i and type-ii slots. in total, there are four intent labels and 14 type-i and four type-ii slot labels. we individually train a text classifier and sequence taggers to perform intent classification and slot filling, respectively. thus, we report the results for collection-related and share-related entities, respectively. ## experiment results tables 2 and 3 present the results for all the experiment models for plue tasks. rows 2-9 show the results of the base plms and their corresponding variants with privacy policy domain-specific continual pre-training. similar to glue @xcite , we also provide the average scores of all plue tasks in the last column of we further investigate the improvement of large variants of plms over base variants of plms on plue tasks. since pp-roberta base performs the best among all base models, we also continue pretrain roberta large (pp-roberta large ). as shown in the last five rows in tables 2 and 3 , the large pre-trained language models mostly outperform their base counterparts. noticeably, pp-roberta large is the best-performing model in app-350, policyqa, pi-extract, and sub-tasks in policyie, and it also achieves the highest average scores of all plue tasks among all models. lastly, even though domain-specific pre-training and large plms help boost the performance for all tasks, the performance of some tasks and datasets (e.g., app-350, privacyqa, slot filling in policyie) remains low, which indicates much potential for further work on nlp for the privacy policy domain. ## related work privacy policy benchmarks the usable privacy policy project @xcite is the most significant effort to date, resulting in a large pool of works @xcite pre-trained language models in the last few years, nlp research has witnessed a radical change with the advent of plms like elmo @xcite and bert @xcite . plms achieved state-of-the-art results in many language understanding benchmarks. consequently, plms have been developed for a wide range of domains, e.g., scientific @xcite , medical @xcite @xcite , legal @xcite , and cybersecurity @xcite . this work investigates the adaptation of plms to facilitate nlp research in the privacy policy domain. ## conclusion and future work reliable aggregation of datasets and benchmarking foundation models on them facilitate future research. this work presents plue, a benchmark for training and evaluating new security and privacy policy models. plue will help researchers benchmark policy language understanding under a unified setup and facilitate reliable comparison. plue also presents some challenges in language understanding evaluation for privacy policies. for example, the imbalance data issue for privacy practices is a major challenge in the priva-cyqa task @xcite . data efficiency is also a challenge for continual pre-training as the amount of unlabeled data is also small for this domain. approaches such as @xcite could be investigated to continually adapt lms for the emerging data in this domain.