Generating Literal and Implied Subquestions to Fact-check Complex Claims (2022)

Abstract: Verifying political claims is a challenging task, as politicians can use various tactics to subtly misrepresent the facts for their agenda. Existing automatic fact-checking systems fall short here, and their predictions like “half-true” are not very useful in isolation, since it is unclear which parts of a claim are true and which are not. In this work, we focus on decomposing a complex claim into a comprehensive set of yes-no subquestions whose answers influence the veracity of the claim. We present CLAIMDECOMP, a dataset of decompositions for over 1000 claims. Given a claim and its verification paragraph written by fact-checkers, our trained annotators write subquestions covering both explicit propositions of the original claim and its implicit facets, such as asking about additional political context that changes our view of the claim’s veracity. We study whether state-of-the-art models can generate such subquestions, showing that these models generate reasonable questions to ask, but predicting the comprehensive set of subquestions from the original claim without evidence remains challenging. We further show that these subquestions can help identify relevant evidence to fact-check the full claim and derive the veracity through their answers, suggesting that they can be useful pieces of a fact-checking pipeline.

## introduction despite a flurry of recent research on automated fact-checking @xcite @xcite @xcite , we remain far from building reliable fact-checking systems @xcite . this challenge motivated us to build more explainable models so the explanations can at least help a user interpret the results figure 1 : an example claim decomposition: the top two subquestions follow explicitly from the claim and the bottom two represent implicit reasoning needed to verify the claim. we can use the decomposed questions to retrieve relevant evidence (section 6), and aggregate the decisions of the sub-questions to derive the final veracity of the claim (section 5.3). @xcite . however, such purely extractive explanations do not necessarily help users interpret a model's reasoning process. an ideal explanation should do what a human-written factcheck does: systematically dissect different parts of the claim and evaluate their veracity. we take a step towards explainable fact-checking with a new approach and accompanying dataset, claimdecomp, of decomposed claims from poli-tifact. annotators are presented with a claim and the justification paragraph written by expert factcheckers, from which they annotate a set of yesno subquestions that give rise to the justification. these subquestions involve checking both the explicit and implicit aspects of the claim (figure 1 ). such a decomposition can play an important role in an interpretable fact verification system. first, the subquestions provide a comprehensive explanation of how the decision is made: in figure 1 , although the individual statistics mentioned by biden ## motivation and task facing the complexities of real-world political claims, simply giving a final veracity to a claim often fails to be persuasive @xcite . to make the judgment of an automatic fact-checking system understandable, most previous work has focused on generating justifications for models' decisions. @xcite ; @xcite ; @xcite used attention weights of the models to highlight the most relevant parts of the evidence, but these only deal with explicit propositions of a claim. @xcite ; @xcite used logic-based systems to generate justifications, yet the systems are often based on existing knowledge graphs and are hard to adapt to complex real-world claims. @xcite treated the justification generation as a summarization problem in which they generate a justification paragraph according to some relevant evidence. even so, it is hard to know which parts of the claim are true and which are not, and how the generated paragraph relates to the veracity. what is missing in the literature is a better intermediate representation of the claim: with more complex claims, explaining the veracity of a whole claim at once becomes more challenging. therefore, we focus on decomposing the claim into a minimal yet comprehensive set of yes-no subquestions, whose answers can be aggregated into an inherently explainable decision. as the decisions to the subquestions are explicit, it is easier for one to spot the discrepancies between the veracity and the intermediate decisions. claims and justifications our decomposition process is inspired by fact checking documents written by professional fact checkers. in the data we use from politifact, each claim is paired with a justification paragraph (see figure 2 ) which contains the most important factors on which the veracity made by the fact-checkers is based. understanding what questions are answered in this paragraph will be the core task our annotators will undertake to create our dataset. however, we frame the claim decomposition task (in the next section) without regard to this justification document, as it is not available at test time. claim decomposition task we define the task of complex claim decomposition. given a claim c and the context o of the claim (speaker, date, venue of the claim), the goal is to generate a set of n yesno subquestions @xmath0, q 2 , ...q n }. the set of subquestions should have the following properties: an individual subquestion should also exhibit: we do not require subquestions to stand alone @xcite ; they are instead interpreted with respect to the claim and its context. ## dataset collection claim / verification document collection we collect political claims and corresponding verification articles from politifact. 3 each article contains one justification paragraph (see figure 2 ) which states the most important factors on which the veracity made by the fact-checkers is based. understanding what questions are answered in this paragraph will be the core annotation task. each claim is classified as one of six labels: pants on fire (most false), false, barely true, half-true, mostly true, and true. we collect the claims from top 50 politifact pages for each label, resulting in a total of 6,859 claims. a claim like "approximately 60,000 canadians currently live undocumented in the usa." hinges on checking a single statistic and is less likely to contain information beyond the surface form. therefore, we mainly focus on studying complex claims in this paper. to focus on complex claims, we filter claims with 3 or fewer verbs. we also filter out claims that do not have an associated justification paragraph. after the filtering, we get a subset consisting 1,494 complex claims. ## automatic claim decomposition the goal is to generate a subquestion set q from the input claim c, the context o, and the target number of subquestions k. ## analyzing decomposition annotations in this section, we study the characteristics of the annotated questions. we aim to answer: (1) how many of the questions address implicit facets of the claim, and what are the characteristics of these? (2) how do our questions differ from previous work on question generation for fact checking @xcite ? (3) can we aggregate subquestion judgments for the final claim judgment? ## 4 did barr vote for legislation that would weaken restrictions for payday lenders? are there any protections for service members using payday lending services? has barr's voting record directly affected protection for veterans against payday lenders? figure 5 : comparison between our decomposed questions with qabriefs @xcite . in general, our decomposed questions are more comprehensive and relevant to the original claim. user study to better quantify the difference, we also conduct a user study in which we ask an annotator to rate how useful a set of questions (without answers) are to determine the veracity of a claim. on 42 claims annotated by both approaches, annotators score sets of subquestions on a likert scale from 1 to 5, where 1 denotes that knowing the answers to the questions does not help at all and 5 denotes that they can accurately judge the claim once they know the answer. we recruit annotators from mturk. we collect 5-way annotation for each example and conduct the t-test over the results. the details can be found in appendix c. ## evidence retrieval with decomposition lastly, we explore using claim decomposition for retrieving evidence paragraphs to verify claims. retrieval from the web to check claims is an extremely hard problem @xcite . we instead explore a simplified proof-of-concept setting: retrieving relevant paragraphs from the full justification document. these articles are lengthy, containing an average of 12 paragraphs, and with distractors due to entity and concept overlap with the claims. we aim to show two advantages of using the decomposed questions: (1) the implied questions contain information helpful to retrieve evidence beyond the lexical information of the claim. (2) we can convert the subquestions to statements and treat them as hypotheses to apply the off-the-shelf nli models to retrieve evidence that entails such hypotheses @xcite . ## related work fact-checking @xcite proposed to decompose the fact-checking process into three components: identifying check-worthy claims, retrieving evidence, and producing verdicts. various datasets have been proposed, including human-generated claims based on wikepedia @xcite @xcite @xcite , real-world political claims @xcite @xcite @xcite , and science claims @xcite . our dataset focuses on real-world political claims, particularly more complex claims than past work which necessitate the use of decompositions. our implied subquestions go beyond what is mentioned in the claim, asking the intention and political agenda of the speaker. @xcite study such implications by gathering expected readers' reactions and writers' intentions towards news headlines, including fake news headlines. to produce verdicts of the claims, other work generates explanations for models' predictions. popat et al. (2017, 2018 2020 ); kotonya and toni (2020) modeled the explanation generation as a summarization task. combining answers to the decomposed questions in our work can form an explicit explanation of the answer. question generation our work also relates to question generation (qg) @xcite , which has been applied to augment data for qa models @xcite @xcite , evaluate factual consistency of summaries @xcite @xcite , identify semantic relations @xcite @xcite , and identify useful missing information in a given context (clarification) @xcite @xcite . our work is most similar to qabriefs @xcite , but differs from theirs in two ways: (1) we generate yes-no questions directly related to checking the veracity of the claim. (2) our questions are more comprehensive and precise. ## conclusion we present a dataset containing more than 1,000 real-world complex political claims with their decompositions in question form. with the decompositions, we are able to check the explicit and implicit arguments made in the claims. we also show the decompositions can play an important role in both evidence retrieval and veracity composition of an explainable fact-checking system. 9 limitations interaction of retrieval and decomposition the evidence retrieval performance depends on the quality of the decomposed questions (compare our results on generated questions to those on annotated questions in section 6). yet, generating high-quality questions requires relevant evidence context. these two modules cannot be strictly pipelined and we envision that in future work, they will need to interact in an iterative fashion. for example, we could address this with a human-inthe-loop approach. first, retrieve some context passages with the claim to verify as a query, possibly focused on the background of the claim and the person who made the claim. this retrieval can be done by a system or a fact-checker. then, we use context passages to retrain the qg model with the annotations we have and the fact-checker can make a judgment about those questions, adding new questions if the generated questions do not cover the whole claim. we envision that such a process can make fact-checking easier while providing data to train the retrieval and qg models.