Automated Justification Production for Claim Veracity in Fact Checking: A Survey on Architectures and Approaches (2024)

Abstract: Automated Fact-Checking (AFC) is the automated verification of claim accuracy. AFC is crucial in discerning truth from misinformation, especially given the huge amounts of content are generated online daily. Current research focuses on predicting claim veracity through metadata analysis and language scrutiny, with an emphasis on justifying verdicts. This paper surveys recent methodologies, proposinga comprehensive taxonomy and presenting the evolution of research in that landscape. A comparative analysis of methodologies and futuredirections for improving fact-checking explainability are also discussed.

## introduction the huge increase in both user-generated and automated content has led to a significant amount of misinformation. this poses risks to uninformed readers, highlighting the need for scalable, automated methods for verification and fact-checking @xcite . while predicting the veracity of claims is essential, relying solely on predictions without providing explanations can be counterproductive, potentially reinforcing belief in false claims and perpetuating misinformation @xcite . most fact-checking models use neural architectures, but interpreting these models is challenging. there is a need for fact-checking frameworks providing justifications to enhance effectiveness and trustworthiness. this survey presents recent efforts addressing automatic justification production for claim verification, emphasizing the move towards "explainable" automated fact-checking (afc). some work refers to the justification production process as the explanation generation process @xcite . in this survey, the term "justification production" is used following the work of @xcite . this survey's main contribution is as follows: firstly, it introduces a multidimensional taxonomy for categorizing works based on various criteria. secondly, it provides how research is progressing towards standard justifications. thirdly, it conducts a comparative analysis of justification production approaches, pipeline architectures, input and output types. lastly, it identifies challenges while proposing future directions in justification production. appendix a outlines the methodology utilized for literature compilation, detailing the search strategy and selection criteria employed for the papers that form the cornerstone of this survey. the survey by @xcite , focusing on justification production, is closely related to ours. however, since then, there has been a significant progress driven by the rapid development of transformer-based architectures and large language models (llms). @xcite only augmented the latter work's taxonomy with counterfactual justifications. while partially covering some recent work, they do not provide a comprehensive, new, detailed multi-dimensional taxonomy as proposed in this survey. ## justification production within afc afc consists of multiple stages forming a pipeline, as shown in figure 1 . one of these stages is justification production. in the upcoming subsections, a brief overview of the general stages in the afc pipeline is provided, with a specific focus on the justification production stage. ## progression towards justifications standardization the aim of justification production is to create a justification that aligns with specific, agreed-upon criteria @xcite , which we refer to as a standard justification. achieving high-quality justifications involves considering certain desired properties known as 'desiderata' @xcite . researchers have collectively agreed upon these desired properties @xcite . producing justifications aligned with these desired properties is crucial for standardizing justifications in explainable afc. @xcite identifies key desiderata for justifications, completeness, where the justification must be valid in full contextuality; coherence, ensuring the faithfulness/consistency between the veracity prediction and justification; interactivity, which is putting into consideration the users' feedback; actionability, providing the user with the needed suggestions for modifying the claim to change it from non-factual to factual; chronology, giving preference to the timing of the claim; novelty, ensuring the justification offers new information; complexity, adjusting the justification's language based on the user's knowledge; parsimony, favouring more short and concise justifications; causality, where a comprehensive causal model is used for deducing causal connections between inputs and the predictions produced. these properties were defined with further details by @xcite , who also added the desideratum of unbiased or impartial justifications. in the context of fact-checking, bias usually manifests as opinions masquerading as evidence. @xcite started the first attempt to provide a standard justification evaluation process by measuring two different types of coherence in the produced justifications: the global coherence which assesses the relevance of a justification in relation to both the claim and its label; and the local coherence which evaluates the cohesion of sentences within a justification. to maintain local coherence, there should be no contradiction between any two sentences in the justification. @xcite started the first attempt to generate standard justifications by adding some desired properties (i.e. faithfulness/coherence, and data consistency) as additional learning signals in the loss function of a transformer-based model @xcite . the data consistency evaluates the similarity of justifications for similar input instances. ## datasets in afc it's worth noting that this survey focuses on providing a new taxonomy, a comparative analysis of justification production approaches, investigating pipeline architectures, addressing challenges encountered, and proposing future directions in afc justification production. comprehensive examinations of datasets were covered thoroughly in previous surveys (mentioned in section 2). however, some information about datasets in afc is also provided in this section. the dataset might contain the needed content for all the stages of the fact-checking pipeline: claim detection, evidence retrieval and selection, veracity verdict production, and justification production. the following paragraphs will discuss the type of content representing each stage with example datasets provided in textual claims are the most common input for fact-checking because they are often produced after the claim detection stage. these claims are usually dataset name claims number veracity verdict justified notes and remarks liar @xcite 12836 yes political dataset statsproperties @xcite 7092 no a knowledge graph dataset scifact @xcite 1409 yes dataset in the scientific domain multifc @xcite 36534 no a multi-domain dataset that also has metadata pubhealth @xcite 11832 yes dataset in the health domain x-fact @xcite 31189 no multi-lingual dataset fever @xcite 185445 no artificially generated dataset hover @xcite 26171 no artificially generated dataset feverous @xcite 87026 no artificially generated dataset and its evidence contains both text and tables klinker @xcite 10000 no a knowledge graph dataset wikifactcheck @xcite 124821 no artificially generated dataset vitaminc @xcite 488904 no artificially generated dataset other sentence-level inputs, such as answers to questions in forums, have also been investigated. some datasets are english, and some are multilingual. metadata, including information such as publication date, sources, and user profiles, is a common type of evidence considered. although metadata can provide complementary insights when textual sources or structural knowledge are lacking, it does not directly substantiate the claim. text-based sources, such as news articles, academic publications, and wikipedia entries, are frequently employed as evidence for fact-checking. multi-modal claims and evidence have recently been researched, and images and videos are considered more credible than text by most audiences. not all the datasets have a binary veracity verdict scheme. fact-checkers often utilize multi-class labels to classify different levels of truthfulness, including categories such as 'true,' 'mostly true,' and 'mixed.' from the perspective of justification production, afc datasets can be classified into two categories: those without justifications and those with justifications. complementing datasets lacking justifications is important. this was done by @xcite on the hover dataset. ## justification production taxonomy multiple dimensions or criteria for categorizing explainable automated fact-checking systems are outlined in this section. we propose five dimensions (illustrated by the first five levels/columns of the taxonomy tree in figure 3 ). the justification process explainability category (section 6.1) indicates whether the process leading to the justification production is explainable or not. then the type of justifications criterion (section 6.2) indicates whether these are a set of sop triples, highlighted parts of selected rationals from the evidence input, or natural language textual justifications. other discriminatory dimensions are the pipeline architecture of the afc components (section 6.3), the input type (section 6.4), whether it is text or multi-modal, and the main approach (section 6.5), which is the categorization of the predominant methods used for justification production. in the following sections, every dimension in the taxonomy is discussed in more details. ## challenges and future directions this section outlines the challenges of producing justifications and highlights promising research efforts to address them, suggesting future directions. evaluating and generating justifications according to desiderata one of the main goals of producing justifications in explainable afc is to align them with specific desiderata (section 4). developing quantitative frameworks, or mathematical formulations, is crucial for measuring desiderata in a structured manner. this allows for a systematic comparison of explainable afc systems based on their incorporation of these desiderata. furthermore, integrating these measurements into the model training process can significantly enhance justification quality. to date, only @xcite has explored modeling and integrating one particular desired property, coherence/faithfulness, as a learning signal in model training. this area has seen limited further exploration. another promising avenue for achieving several desiderata, given the recent proliferation of rea-soners with llms (section 6.5.4) and multihop (section 6.5.5), could be the production of counterfactual justifications, as suggested by @xcite and illustrated in figure 4 . counterfactual justifications involve imagining scenarios or outcomes that did not actually occur and exploring their consequences. they involve alternate scenarios -for example, 'if the earth were flat, we would be able to see boats even when they are very far away using telescopes' (as shown in figure 4 ). when incorporated into the justifications, counterfactual reasoning can reinforce some desired properties such as completeness and coherence. moreover, some desiderata not fully achieved by current work @xcite , like actionability, could also be realized. for instance, counterfactual justifications can identify specific elements in a claim that, if altered, could render it factual, thus guiding users toward more accurate statements. by offering alternative perspectives on a claim, counterfactual justifications can provide novelty via new information that might not be apparent through traditional justification methods. counterfactual justifications inherently involve understanding causal relationships, another highly desired property. the ultimate objective remains to incorporate most or all of the desiderata presented in section 4. ## conclusion in summary, this survey contributes a novel multidimensional taxonomy, comprehensively presents the architectures employed in justification production, explores emergent methodologies, conducts a comparative analysis of these methodologies, and proposes prospective avenues for further research.