Cross-Policy Compliance Detection via Question Answering (2021)

Abstract: Policy compliance detection is the task of ensuring that a scenario conforms to a policy (e.g. a claim is valid according to government rules or a post in an online platform conforms to community guidelines). This task has been previously instantiated as a form of textual entailment, which results in poor accuracy due to the complexity of the policies. In this paper we propose to address policy compliance detection via decomposing it into question answering, where questions check whether the conditions stated in the policy apply to the scenario, and an expression tree combines the answers to obtain the label. Despite the initial upfront annotation cost, we demonstrate that this approach results in better accuracy, especially in the cross-policy setup where the policies during testing are unseen in training. In addition, it allows us to use existing question answering models pre-trained on existing large datasets. Finally, it explicitly identifies the information missing from a scenario in case policy compliance cannot be determined. We conduct our experiments using a recent dataset consisting of government policies, which we augment with expert annotations and find that the cost of annotating question answering decomposition is largely offset by improved inter-annotator agreement and speed.

## introduction policy compliance detection is the task of ensuring that a scenario conforms to a policy. it is a task that occurs in various contexts, e.g. in ensuring correct application of government policies @xcite , enforcing community guidelines on social media platforms @xcite or the resolution of legal cases @xcite , etc. while the task can be modelled as a text-pair classification similar to textual entailment @xcite to 13 weeks in advance? q1: is it up to 17 weeks in advance? q2: are you aged 60 or over? q3: are you moving? ➤ claiming in advance and backdating and or and q3 q0 q1 q2 you can claim in advance by up to 13 weeks (or 17 weeks if you're aged 60 or over), if you're moving. you won't usually get any money before you move. scenario: my partner and i will be moving shortly. i had my 40th birthday last week. et al. @xcite , i.e. whether a scenario complies with a certain policy, this often results in poor accuracy due to the complexity of the policy descriptions, which often contain multiple clauses connected with each other @xcite . moreover, even though some types of policy compliance detection can be tackled via supervised classification, e.g. recognizing hateful memes @xcite and other forms of hate speech or abusive language on social platforms @xcite , this requires substantial amounts of training data for one specific policy which is expensive and time-consuming. furthermore, the requirement for policy-specific training data renders supervised classification difficult to adapt when policies change, e.g. customs rules changing with brexit, novel forms of hate speech in the context of the covid pandemic, etc. in this paper we propose to address policy com-pliance detection via decomposing it to question answering (qa) and an expression tree. each policy description is converted into a set of questions whose answers are combined in a logical form represented by the expression tree to produce the label. this allows for better handling of complex policies, and to take advantage of the large-scale datasets and models developed for question answering @xcite . in addition, the questioning of the scenario with respect to parts of the policy identifies which parts were used in order to determine compliance and which information is missing in case the scenario could not be resolved. this in turn, enhances the interpretability of the model's decisions and can identify areas for improvement. figure 1 shows an example application of our approach in the context of a government policy where the missing date of the move in the scenario results in its compliance with the policy being unresolved. we demonstrate the benefits of policy compliance detection via qa using a dataset that contains policies, decomposition questions and expression trees and scenarios. while the policies and scenarios in the dataset are taken from sharc @xcite , we augment them with expression trees and answers to each question for all the scenarios and policies to create the question answering for policy compliance (qa4pc ) dataset. the results of our experiments demonstrate that we can achieve an accuracy of 0.69 for policies unseen during the training (an increase of 25 absolute points over the entailment approach) and an accuracy of 0.59 (an increase of 22 absolute points) when no in-domain data is available for training. we also show that our approach is more robust compared to entailment when faced with policies of increasing complexity. in addition, we find that the cost of annotating question answering decomposition and expression trees is largely offset by improved interannotator agreement and speed. finally, we release the qa4pc dataset to the community to facilitate further research. 2 ## previous work on policy compliance policy compliance as an entailment task has been studied by @xcite and referred to as statutory reasoning entailment. they find that the application of machine reading models exhibits low performance, whether or not they have been 2 please contact the authors to access the dataset. similarity search or matching has been used to match a new post to known violating posts on social media such as hoaxes, an objectionable video or image, or a hateful meme @xcite @xcite . for textual content, this can be compared to an entailment task. this approach requires a bank of existing violating content for each policy. by using the policy description and decomposing it into several qa tasks, breaches for new policies can be detected, as we show in our experiments. much work has focused on learning important elements of privacy policies to assist users in understanding the terms of the policies devised by different websites. @xcite uses question answering to extract different elements of privacy policies that are informative to the users. nejad et al. (2020) and mustapha et al. (2020) assign pre-defined categories to privacy policy paragraphs using supervised classification. while these works aim to help users in understanding complex text of privacy policies, they do not aim to identify compliance and they mainly focus on privacy policies. in our work, we look at a wide range of government policies, for different governments and focus on an approach for cross-policy compliance detection. ## policy compliance detection via qa in this section we describe our proposed approach for policy compliance detection via question answering. first we formulate the task as textual entailment following previous work @xcite . then we describe how our proposed expression trees combined with qa operate. finally we discuss the implications of our approach for training and data annotation requirements. ## data in this section we discuss the creation of the question-answering for policy detection (qa4pc) dataset which we introduce in this work to support our experiments. we create qa4pc using the policies and the scenarios from the publicly available training and development splits of existing dataset sharc @xcite . a scenario in sharc is a real-world situation described by a user who is conversing with a system to find out whether they comply with a policy. each conversation utterance in sharc has a policy, a question and an answer. an utterance may include a conversation history (a list of qas) and/or a scenario which is built from a conversation list (a list of questions and answers). the answer to an utterance can be yes/no or a follow-up clarification question. 3 however, while sharc conversations contain some of the questions related to the policies, we found that many were missing, as the conversation progress may render answers to them irrelevant. a such example is provided in figure 6 in the appendix. details on number of qas per policy for both datasets are discussed in section 4.4. as neither entailment-style classification nor the question answering expression trees require complete question-answer sets per instance or expression trees for training, we convert 70% of sharc policies for training of entailment and qa tasks (see section 4.1 for more details). the procedure for converting the data is explained in the next section. in order to ensure the evaluation datasets (dev and test sets) are correctly annotated, we augment the remaining 30% of sharc policies with questions and expression trees, and the scenarios with the entailment labels and answers to all questions of the related policy (see section 4.2). two annotators, co-authors of this paper and uk nationals, were involved in the annotation described in this section. ## experimental setup model we use roberta @xcite for both the entailment baseline and the qa subtask and learn three-way classifiers for each. we used the implementation in the huggingface (wolf et al., 2020) library. we also experimented with a t5 model @xcite on the dev set. since the results of t5 model were very similar to roberta, we report the performance on all the tasks using roberta. entailment we use a cross-encoder set up for the task. for each instance, we combine the policy, scenario and the question using the following format: "premise: [policy] sep hypothesis: [scenario] cls". the embeddings of the token cls is used to classify the instance. this is a common approach for entailment modelling using pre-trained encoders @xcite @xcite . question answering for pcd in the qa approach to pcd, we decompose the task into a qa subtask and expression trees, where we use the expression tree of a policy to combine the answers to the questions in the policy to infer the label. a qa model is trained on qa instances of the dataset. for each qa instance, we combine the scenario and the question using the format: "passage: [scenario] sep question: [question] cls". this is a common formulation of qa tasks @xcite using transformer models. the embedding of the cls token is used to assign a label to each instance. ## discussion assumptions in this work, we assume that we can decompose policy descriptions into independent questions that can be executed in parallel. however, designing independent questions is not always straightforward. e.g. it could be useful to have a question in an expression tree that is a follow-up on another question and therefore a qa model will have to be executed sequentially taking into account the earlier question in formulating the follow up one. ## conclusion in this work, we propose to address the task of policy compliance detection via decomposing a policy description into an expression tree. the expression tree consists of a set of questions and a logical expression that combines the questions to infer a final label. our experiments show that compared to the existing entailment approach @xcite , qa decomposition results in a better model accuracy in a cross-policy setting using indomain (0.69 vs. 0.44) and out-of-domain (0.59 vs. 0.37) training data. furthermore, we show that while there is an upfront cost of annotation for expression trees, the cost of qa decomposition annotations is lower than the cost of annotating the entailment task while reaching a higher agreement. future work can investigate ways to generate expression trees for policy descriptions automatically. further, it will be beneficial to demonstrate that this approach is suitable in detecting policy compliance in other domains such as community standards implemented by social media platforms which requires annotation of relevant datasets.