Varifocal Question Generation for Fact-checking (2022)

Abstract: Fact-checking requires retrieving evidence related to a claim under investigation. The task can be formulated as question generation based on a claim, followed by question answering.However, recent question generation approaches assume that the answer is known and typically contained in a passage given as input,whereas such passages are what is being sought when verifying a claim.In this paper, we present Varifocal , a method that generates questions based on different focal points within a given claim, i.e. different spans of the claim and its metadata, such as its source and date.Our method outperforms previous work on a fact-checking question generation dataset on a wide range of automatic evaluation metrics.These results are corroborated by our manual evaluation, which indicates that our method generates more relevant and informative questions.We further demonstrate the potential of focal points in generating sets of clarification questions for product descriptions.

## introduction the growing amount of information online and its impact have increased the need for fact-checking, i.e. judging whether a claim is true or false. to determine the truthfulness of a claim, fact-checkers need to answer questions related to the claim, world knowledge within its time frame, local politics, etc. @xcite . using questions and answers has also been shown to be an effective way of conveying fact-checks. for instance, @xcite found that presenting information related to covid-19 as answers to questions improved attitudes towards vaccination more than merely presenting the relevant facts. as professional fact-checkers can spend a day to verify a single claim depending on its complexity @xcite , there * equal contribution. previous work on question generation assumes that the answer is known, typically contained in a passage given in the input @xcite @xcite . such passages, though, are what is being sought when factchecking a claim. the only exception is recent work on clarification questions @xcite . however, work in this area examines a specific narrow domain where a limited number of questions can be asked, e.g. questions related to product descriptions in the amazon dataset of @xcite , or dialogues in the ubuntu dataset @xcite , which is not the case in fact-checking. factchecking questions are more diverse and may rely on the experience and intuition of the fact-checker as they aim to scrutinize every piece of information within the claim. they can be as generic as questions about general definitions or as specific as those tackling details about a one-time event. in this paper, we propose an approach that generates questions for claim verification, which we name varifocal. it uses focal points from different spans of the claim as well as its metadata, i.e. its source, and its date. each focal point guides the generator to question a different part of the claim; e.g. in figure 1 when "miss universe guyana 2017" is used as the focal point, the question generated is about who she is. we evaluate our approach on the qabriefs dataset introduced by @xcite using a wide range of automatic metrics, and show that figure 1 : the architecture of varifocal. we use a dependency parser to extract the different focal points, i.e. spans, then generate questions based on them. we rank the generated questions using a re-ranker and return the top n questions. the example in the figure was generated by our system. we show three highlighted focal points along with the (output) questions they led to. varifocal performs the best among the different systems considered. in addition, we conduct a human evaluation on questions generated by four different systems and gold standard questions based on four criteria: a) intelligibility, b) clarity, c) relevance, and d) informativeness. the results show that varifocal generates more intelligible, clear, relevant, and informative questions than the other systems, corroborating the results of the automatic evaluation. * finally, we apply varifocal to generating sets of clarification questions on amazon product descriptions @xcite , where it shows competitive performance against methods that generate single questions while having other ones in the set as part of the input @xcite . ## related work the main piece of previous work on question generation for fact-checking is by @xcite . they proposed the qabriefs dataset, which consists of claims with manually annotated question-answer pairs containing additional information about the claims (e.g. the exact definition of a term, the content of a bill, details about a political statement or a vote). the qabriefs dataset contained questions asked by crowd-workers, who had to read both the claim and its fact-checking article. fan et al. ( 2020 ) presented the qabriefer model, which generates a set of questions conditioned on the claim, searches the web for evidence and retrieves answers. however, they evaluated the questions generated only using bleu scores without conducting a human evaluation. more recently, @xcite addressed the problem of explainability in fact-checking through question answering using the fool me twice corpus @xcite . they generated questions from the claim, retrieved answers from the evidence, and compared them to the generated ones. yet, they did not evaluate their question generation process, and assumed that the evidence is given as input to generate the questions, which is unrealistic since the questions are typically used for evidence retrieval. other related work includes @xcite who introduced a dataset containing 32k instances of real-world policies, crowd-sourced fictional life scenarios, and dialogues in order to reach a final yes/no answer. the policies were given as input and were explicitly stating what information needed to be asked for, and the questions had to have a yes or no answer. neither of these hold in fact-checking, where questions are not usually answered by yes or no, and the information to be searched for is not known in advance. @xcite presented a method to generate clarification questions. they built a two-stage framework that identifies missing information using the notions of global and local schemas. the global schema was built using filtered key phrases extracted from contexts that were part of the same class of the data, e.g. a class of similar products in the amazon data @xcite and similar dialogues in the ubuntu dataset @xcite , whereas the local schema was built using one given context, and they defined the missing information as the difference between the global and the local schema. the extraction of comparable schemas across different contexts was possible due to the repetitive nature of the datasets considered, e.g. the descriptions of products of the same type such as laptops allow the prediction of potentially missing properties which need clarification, in contrast to fact-checking claims which are less repetitive. the standard sequence-to-sequence architecture @xcite is typically used in question generation approaches @xcite . although answer-aware approaches allow for the generation of multiple questions conditioned on the same passage @xcite , providing the answer during inference is not possible in fact-checking since one would typically ask questions about what is missing from the claim. other work includes question generation for question answering @xcite , question generation for educational purposes @xcite , and poll question generation from social media posts @xcite . furthermore, @xcite evaluated rewards in question generation, showed that they did not correlate with human judgments, and explained why rewards did not help when using reinforcement learning. commonly used evaluation metrics such as bleu @xcite and rouge @xcite ) fall short at correlating with human judgments when evaluating the quality of automatically generated questions @xcite @xcite . @xcite carried a human evaluation based on fluency, relevance, whether the question dealt with missing information, and usefulness. in addition, @xcite proposed to assess the quality of automatically generated questions based on whether they were well-formed, concise, answerable, and answer-matching. similarly, we conduct a human evaluation of the generated questions adapted to fact-checking. ## varifocal question generation in this section, we describe varifocal, an approach that generates multiple questions per claim based on its different aspects, which correspond to textual spans that we call focal points. varifocal consists of three components: (1) a focal point extractor, (2) a question generator that generates a question for each focal point, and (3) a re-ranker that ranks the generated questions, removes duplicates and promotes questions that are more likely to match the gold standard ones. ## analysis to investigate the quality of the different focal points and how they guide the question generation process (1) we extracted focal points that are named entities only and those that are not, then automatically evaluated the performance of varifocal on each of these subsets; when analysing the results, we observed that focal points whose syntactic roles are nsubj, dobj, pobj, prep and compound seem to lead to the best questions. we, therefore, examined their average informativeness scores according to the human raters. as shown in ## limitations and future work the complexity of the claims in our experiments on the qabriefs dataset, the maximum number of focal points was 175, as shown in ## conclusion we introduced varifocal, a question generation method for fact-checking that alleviates the absence of full context using focal points. varifocal, with and without metadata, improves the quality of automatically generated questions compared to other systems. it generates intelligible and clear questions, and sometimes questions that can be more relevant and informative than the gold standard ones. moreover, when we used varifocal to generate sets of clarification questions, it showed comparable results to those achieved by models that generate single questions while having additional ones as part of the input. in the future, we will consider extensions using additional knowledge related to the different aspects of the claim and assess the overall quality of sets of questions.