FACTIFY -5 WQA : 5 W Aspect-based Fact Verification through Question Answering (2023)

Abstract: Automatic fact verification has received significant attention recently. Contemporary automatic fact-checking systems focus on estimating truthfulness using numerical scores which are not human-interpretable. A human fact-checker generally follows several logical steps to verify a verisimilitude claim and conclude whether it’s truthful or a mere masquerade. Popular fact-checking websites follow a common structure for fact categorization such as half true, half false, false, pants on fire, etc. Therefore, it is necessary to have an aspect-based (delineating which part(s) are true and which are false) explainable system that can assist human fact-checkers in asking relevant questions related to a fact, which can then be validated separately to reach a final verdict. In this paper, we propose a 5W framework (who, what, when, where, and why) for question-answer-based fact explainability. To that end, we present a semi-automatically generated dataset called FACTIFY-5WQA, which consists of 391, 041 facts along with relevant 5W QAs – underscoring our major contribution to this paper. A semantic role labeling system has been utilized to locate 5Ws, which generates QA pairs for claims using a masked language model. Finally, we report a baseline QA system to automatically locate those answers from evidence documents, which can serve as a baseline for future research in the field. Lastly, we propose a robust fact verification system that takes paraphrased claims and automatically validates them. The dataset and the baseline model are available at https: //github.com/ankuranii/acl-5W-QA

## fact checking demands aspect-based explainability manual fact-checking is a time-consuming task. to assess the truthfulness of a claim, a journalist would either need to search online, offline, or both, brows- † work does not relate to the position at amazon. there are several contemporary practices that journalists use for the manual verification of a claim. these methods can be categorized into four broad categories @xcite : research on automatic fact-checking has recently received intense attention @xcite factify question answering at a glance entailment classes textual support no. of claims no. of paraphrased claims 5wqa pairs no. of evidence documents support text are supporting each other ∼ similar news 217,856 992,503 464,766 217,635 neutral text are neither supported nor refuted ∼ may have common words 79,318 365,593 194,635 45,715 refute fake claim 93,867 383,035 243,904 93,766 total 391,041 1,741,131 903,305 357,116 when the covid-19 vaccines were in work? ans: before pandemic. contemporary automatic fact-checking systems focus on estimating truthfulness using numerical scores which are not human-interpretable @xcite . others extract explicit mentions of the candidate's facts in the text as evidence for the candidate's facts, which can be hard to spot directly. moreover, in the case of false information, it is commonplace that the whole claim isn't false, but some parts of it are, while others could still be true. a claim is either opinion-based, or knowledge-based @xcite . for the same reason, the popular website politifact based on the work by (garg and sharma, 2020) categorized the fact-checking verdict in the form of half-true, half-false, etc. we propose 5w (who, what, when, where, and why) aspect-based question-answer pairwise explainability. including these 5w elements within a statement can provide crucial information regarding the entities and events being discussed, thus facilitating a better understanding of the text. for instance, in the statement "moderna's lawsuits against pfizer-biontech show covid-19 vaccines were in the works before the pandemic started." the use of who highlights the individuals or entities involved in the action of filing lawsuits, what pertains to the content of the lawsuit, specifically the revelation that covid-19 vaccines were in the works, when refers to the timing of this revelation, i.e., before the pandemic. overall, the incorporation of "who," "what," "when," "where," and "why" in a text can provide crucial context and aid in making the text more clear and comprehensible. automatic question and answering (q&a) systems can provide valuable support for claims by providing evidence and supporting information. they can also help to identify potential flaws or weaknesses in a claim, allowing for further analysis and discussion. they can also help to identify potential flaws or weaknesses in a claim, allowing for further analysis and discussion. only two recent works @xcite propose question answering as a proxy to fact verification explanation, breaking down automated fact-checking into several steps and providing a more detailed analysis of the decision-making processes. questionanswering-based fact explainability is indeed a very promising direction. however, open-ended qa for a fact can be hard to summarize. therefore, we refine the qa-based explanation using the 5w framework (who, what, when, where, and why). journalists follow an established practice for fact-checking, verifying the so-called 5ws @xcite , @xcite , @xcite , @xcite , @xcite . this directs verification search and, moreover, identifies missing content in the claim that bears on its validity. one consequence of journalistic practice is that claim rejection is not a matter of degree (as conveyed by popular representations such as a number of pinocchios or crows, or true, false, half true, half false, pants on fire), but the rather specific, substantive explanation that recipients can themselves evaluate @xcite . ## data sources and compilation data collection is done by sorting 121 publicly available prevalent fact verification data sets based on modalities (111), languages (83), and tasks (51). vitc fever hover factify 1.0 factify 2.0 faviq (175905) (135545) (21000) (15555) 34.66% 0% 100% 44.98% total: 391041 5.37% 3.97% (22036) 5.63% by filtering 121 publicly available data sets for fact verification, we found ten of them to be suitable for the text-based fact verification task. we only considered the claims present in textual format in english-language because of which danfever @xcite and x-fact @xcite were also excluded because they are either danish or multilingual. we discovered that "evidence-based factual error correction" and feverous @xcite were subsets of the fever dataset, so we decided to use fever @xcite , hover @xcite , vitc @xcite , faviq @xcite , factify 1.0 @xcite and factify 2.0 @xcite for our analysis. we verified that the claims in these datasets were unique but found that 64 claims from vitc @xcite overlapped with those in fever @xcite which is later considered once giving a total count of 391, 041 datapoints and the distribution is represented in the figure 2 . we only used a specific number of claims from each of the six datasets after manually inspecting the quality aspects -length of the claim and evidence, grammatical correctness, etc. for the fever and vitc datasets, only the claims belonging to the train split were used for making the dataset. for factify 1.0 and factify 2.0, the multimodal part of the dataset was discarded and only the text-based part was used. faviq has two sets: the a set and the r set. a set consists of ambiguous questions and their disambiguation. r set is made by using unambiguous question-answer pairs. as discussed in earlier paragraphs, a set is a more challenging set; hence we took the a set of faviq for making our dataset. in the case of the hover dataset, 22036 claims were used in making our dataset. we propose an amalgamated data set with the total number of unique claims as 391, 041. around (∼ 85%) of them are from vitc, fever, and hover, and (∼ 15%) of it is from factify 1.0, factify 2.0 and faviq as evident from figure 2. figure 1 offers a snapshot of topics in these datasets through a word cloud. ## paraphrasing textual claims the motivation behind paraphrasing textual claims is as follows. a textual given claim may appear in various different textual forms in real life, owing to variations in the writing styles of different news publishing houses. incorporating such variations is essential to developing a strong benchmark to ensure a holistic evaluation (see examples in figure 3 ).manual generation of possible paraphrases is undoubtedly ideal, but that process is time-consuming and labor-intensive. on the other hand, automatic paraphrasing has received significant attention in recent times @xcite ) @xcite (witteveen and andrews, 2019) @xcite . for a given claim, we generate multiple paraphrases using various sota models. in the process of choosing the appropriate paraphrase model based on a list of available models, the primary question we asked is how to make sure the generated paraphrases are rich in diversity while still being linguistically correct. we delin- moderna's lawsuits against pfizer-biontech show covid-19 vaccines were in the works before the pandemic started. prphr 1: moderna's legal action against pfizer-biontech demonstrates that work was being done on covid-19 vaccines prior to the outbreak of the pandemic. prphr 2: moderna's legal action against pfizer-biontech implies that work on covid-19 vaccines had begun prior to the beginning of the pandemic. prphr 3: moderna's court cases against pfizer-biontech indicate that covid-19 vaccines had been in development before the pandemic began. prphr 4: moderna's prosecution against pfizer-biontech demonstrates that covid-19 vaccines had been in advancement prior to the pandemic commencing. prphr 5: it is revealed by moderna's legal actions addressed to pfizer-biontech that work on covid-19 vaccines was being done before the pandemic began. figure 3 : claims and paraphrases obtained using text-davinci-003 @xcite eate the process followed to achieve this as follows. let's say we have a claim c. we generate n paraphrases using a paraphrasing model. this yields a set of p c 1 , . . ., p c n . next, we make pair-wise comparisons of these paraphrases with c, resulting in cp c 1 , . . ., and cp c n . at this step, we identify the 10424 examples which are entailed, and only those are chosen. for the entailment task, we have utilized roberta large @xcite -a sota model trained on the snli task @xcite . however, there are many other secondary factors, for e.g., a model may only be able to generate a limited number of paraphrase variations compared to others, but others can be more correct and/or consistent. as such, we considered three major dimensions in our evaluation: (i) a number of considerable paraphrase generations, (ii) correctness in those generations, and (iii) linguistic diversity in those generations. we conducted experiments with three available models: (a) pegasus @xcite , (b) t5 (t5-large) @xcite , and (c) gpt-3 (text-davinci-003 variant) @xcite . based on empirical observations, we concluded that gpt-3 outperformed all the other models. to offer transparency around our experiment process, we detail the aforementioned evaluation dimensions as follows. coverage -a number of considerable paraphrase generations: we intend to generate up to 5 paraphrases per given claim. given all the generated claims, we perform a minimum edit distance (med) @xcite ) -units are words instead of alphabets). if med is greater @xmath0.g., cp c 1 ) with the claim, then we further consider that paraphrase, otherwise discarded. we evaluated all three models based on this setup that what model is generating the maximum number of considerable paraphrases. correctness -correctness in those generations: after the first level of filtration we have performed pairwise entailment and kept only those paraphrase candidates, are marked as entailed by the @xcite (roberta large), sota trained on snli @xcite . diversity -linguistic diversity in those generations: we were interested in choosing that model can produce linguistically more diverse paraphrases. therefore we are interested in the dis-similarities check between generated paraphrase claims. for e.g., cp c n , p c 1p c n , p c 2p c n , . . . , p c n-1p c n and repeat this process for all the other paraphrases and average out the dissimilarity score. there is no such metric to measure dissimilarity, therefore we use the inverse of the bleu score @xcite . this gives us an understanding of how linguistic diversity is produced by a given model. based on these experiments, we found that text-davinci-003 performed the best. the results of the experiment are reported in the following table. furthermore, we were more interested to choose a model that can maximize the linguistic variations, and text-davinci-003 performs on this parameter of choice as well. a plot on diversity vs. all the chosen models is reported in figure 4 . ## 5w semantic role labelling claim. moderna's lawsuits against pfizer-biontech show covid-19 vaccines were in the works before the pandemic started. verbs. the five "w"s (what, when, where, why, who) are often used to refer to the key questions that need to be answered in order to fully understand a sentence or piece of text. srl can be seen as a way of providing answers to these questions by identifying the various roles that words and phrases play within a sentence. for example, a semantic role labeler might identify the subject of a sentence (who or what the sentence is about), the object (who or what is being acted upon), and the verb (the action being performed). in this way, semantic role labeling can be seen as a way of providing the necessary context for answering the five "w"s, and can be an important tool in natural language processing and understanding. in this study, we use the mapping displayed in after the mapping is done, a detailed analysis for the presence of each of the 5w is conducted which is summarized in figure 6 . as indicated in figure 5 , the pipeline for generating 5w aspect-based semantic role labeling is to pass it through an srl model and map it with 5w. an example of a claim as per the output using allennlp's srl model is in figure 5 . ## selecting the best combination -5w qag vs. 5w qa validation we have utilized off-the-self models both for 5w question-answer generation and 5w questionanswer validation. given that the datasets used for training the models bear an obvious discrepancy in terms of the distribution characteristics compared to our data (world news) which would probably lead to a generalization gap, it was essential to experimentally judge which system offered the best performance for our use-case. instead of choosing the best system for generation vs. validation, we opted for pair-wise validation to ensure we chose the best combination. ## conclusion and future avenues it has been realized by the community that due to the given complexity of fact-checking it possibly can not be automated completely. human-inloop is the solution for the same. proposed 5w qa-based fact verification can be the best aid for human fact-checkers. to the best of our knowledge, we are the first to introduce 5w qa-based fact verification and additionally proposed relevant techniques to automatically generate qa using the automatic method, which can be readily used for any incoming claim on the spot. furthermore, the qa validation section can aid to provide evidence support. paraphrasing claims provide a holistic approach to fact-checking. generated datasets and resources will be made public for research purposes containing 3.91 million claims.