Retrieval Enhanced Data Augmentation for Question Answering on Privacy Policies (2023)

Abstract: Prior studies in privacy policies frame the question answering (QA) task as identifying the most relevant text segment or a list of sentences from a policy document given a user query. Existing labeled datasets are heavily imbalanced (only a few relevant segments), limiting the QA performance in this domain. In this paper, we develop a data augmentation framework based on ensembling retriever models that captures the relevant text segments from unlabeled policy documents and expand the positive examples in the training set. In addition, to improve the diversity and quality of the augmented data, we leverage multiple pre-trained language models (LMs) and cascaded them with noise reduction oracles. Using our augmented data on the PrivacyQA benchmark, we elevate the existing baseline by a large margin (10% F1) and achieve a new state-of-the-art F1 score of 50%. Our ablation studies provide further insights into the effectiveness of our approach.

## introduction privacy policies describe how service providers collect, manage, and use their users' data. understanding them is crucial for users as they can determine if the conditions outlined are acceptable. policy documents, however, are lengthy, verbose, equivocal, and hard to understand @xcite . consequently, they are often ignored and skipped by users @xcite . building question answering (qa) systems for privacy policies is a stepping stone to allow users to ask questions about their rights. prior works @xcite framed the qa task as a sentence selection task, essentially a binary classification task that identifies if a policy text segment is relevant to a question. since policy documents consist of many sentences and typically a few are relevant to a question, 1 the 1 privacyqa @xcite dataset has 1,350 segmented policy document s (s 1 ) we do not sell or rent your personal information to third parties for their direct marketing purposes without your explicit consent. (s n ) ...we will not let any other person, including sellers and buyers, contact you, other than through your ... queries i annotating the red segment as irrelevant (i 1 ) how does fiverr protect freelancers' personal information? (i 2 ) what type of identifiable information is passed between users on the platform? queries r annotating the red segment as relevant (r 1 ) what are the app's permissions? (r 2 ) what type of permissions does the app require? queries d that annotators disagree about relevance (d 1 ) do you sell my information to third parties? (d 2 ) is my information sold to any third parties? we evaluate our framework on the privacyqa benchmark. we elevate the state-of-the-art performance significantly (10% f1) and achieve a new one (50% f1). furthermore, our ablation studies provide an insightful understanding of our model. we will release all data and code upon acceptance. ## methodology the privacy policy qa is a binary classification task that takes a user query q, a sentence p from policy documents and outputs a binary label @xmath0, 1} that indicates if q and p are relevant or not. as most sentences p are labeled as negative, our goal is to retrieve relevant sentences to augment the training data and mitigate the data imbalance issue. given a 2 to verify, in our preliminary study, for each positive example in the privacyqa training set, we augment a new synthesized positive example by en-zh-en back-translation using google translator api, and the performance drops by 3% f1. 3 we refer the misclassified candidates as noisy retrievals. qa training dataset @xmath1, p i , z i )} m @xmath2, for each question in d, we (1) retrieve positive sentences from a large unlabeled corpus. (2) filter the noisy examples using filter models and aggregate final candidates. the final candidates are combined with the base data d to train the end qa model. we use an ensemble of retrievers and filter models built upon various pre-trained lms throughout the whole process. details are discussed in the following. retriever. our retriever module is built upon the dense passage retriever (dpr) model @xcite . it consists of two encoders q(⋅) and p (⋅) that encode the queries and the policy sentences, respectively. the relevance of a query q and a policy sentence p is calculated by the dot product of q(q) and p (p), i.e., sim(q, p) = q(q) t ⋅ p (p). we train a retriever r l on d, where the encoders in r l are initialized with a pre-trained lm l. at inference, r l retrieves the top-k most relevant policy sentences from an unlabeled corpus of pol- ## conclusion we develop a noise-reduced retrieval-based data augmentation method that combines different pretrained language models to address the data imbalance issue in privacy policy qa. however, our approach can possibly be adapted to other domains and we leave the exploration as the future work.