AMRE x: AMR for Explainable Fact Verification (2024)

Abstract: With the advent of social media networks and the vast amount of information circulating through them, automatic fact verification is an essential component to prevent the spread of misinformation. It is even more useful to have fact verification systems that provide explanations along with their classifications to ensure accurate predictions. To address both of these requirements, we implement AMREx, an Abstract Meaning Representation (AMR)-based veracity prediction and explanation system for fact verification using a combination of Smatch, an AMR evaluation metric to measure meaning containment and textual similarity, and demonstrate its effectiveness in producing partially explainable justifications using two community standard fact verification datasets, FEVER and AVeriTeC. AMREx surpasses the AVeriTec baseline accuracy showing the effectiveness of our approach for real-world claim verification. It follows an interpretable pipeline and returns an explainable AMR node mapping to clarify the system’s veracity predictions when applicable. We further demonstrate that AMREx output can be used to prompt LLMs to generate natural-language explanations using the AMR mappings as a guide to lessen the probability of hallucinations.

## introduction with the vast amount of information circulating on social media and the constantly changing claims about various topics, automatic fact verification has become crucial for preventing the spread of misinformation. to address this need, automatic fact-checking task @xcite and several shared tasks have been introduced to encourage nlp researchers to develop systems that gather evidence (fact extraction) for a given claim and classify it (fact verification) as to its predicted veracity. examples include fever @xcite and the current averitec task @xcite , which employ the labels supports, refutes, notenoughinfo (nei) or conflictingevidence/cherrypicking. natural language inference (nli) systems, which assess whether a premise semantically entails a given hypothesis @xcite , have been used for fact verification, yielding demonstrably strong results in the fever shared task. however, there has been limited focus on the explainability of these implementations. recent studies @xcite have highlighted nli models' tendency to rely on spurious cues for entailment classification making it important to provide clear explanations alongside fact verification predictions. we design and implement a new, deterministic nli system based on abstract meaning representation (amr), dubbed amrex, and test it on the fever and averitec fact-checking datasets. amr is a rooted, directed, acyclic graph with nodes representing concepts and edges denoting the relations @xcite . this representation captures semantic relationships among entities that can be difficult to identify in a syntactic representation @xcite . we apply an existing amr evaluation metric @xcite , to map claims (e.g., x was produced y) to relevant evidence (e.g., x is a film produced by y). we incorporate this mapping into our amrex system to yield partially explainable fact verification. we assume evidence collection has already been completed, as our focus is on the potential for explainability of our fact-checking results, independent of the degree of correctness with respect to a ground truth. this, in fact, is the key contribution of this paper: we demonstrate that explainability is valuable regardless of performance levels. if performance is high, explainability supports an exploration of the factors contributing to the algorithm's success. if performance is low, it serves as a diagnostic tool to understand what went wrong. fig. 1 illustrates our explainable output using claim: veeram was produced by vijaya productions. evidence: veeram (valour) is a 2014 indian tamil action film directed by siva and produced by vijaya productions. amrex supports amr mapping explanation amr graph mapping. we quantify the degree to which the claim amr is contained in the evidence amr and present the mappings identified in this process to demonstrate whether the claim is embedded within the evidence. for example, the claim veeram was produced by vijay productions and evidence veeram (valour) is a 2014 indian tamil action film directed by siva and produced by vijaya productions are represented as amrs and processed through the smatch algorithm. this identifies similar substructures between them, showing that both texts mention a production (rooted by produce-01 predicate) with similar attributes and refer to the same film (through substructures rooted by work-of-art and film in the claim and evidence amrs). amrex uses this high-level notion of meaning containment, along with a textual similarity score, to produce the veracity prediction "supports". section 2 reviews existing nli implementations and explainable representations used in fact verification. section 3 provides a detailed description of amrex system and the experiments conducted. section 4 presents an analysis and discussion of the results, with conclusions in section 5. ## related work below we explore existing studies related to nli for fact verification, explainable representation of fact verification, and amr. ## experiment this section presents the details behind datasets used in our experiments, along with the experimental steps carried out to build amrex model. ## results and analyses we experiment with λ values in the [0,1] range for eq. 1 on both fever and averitec datasets to find the best combination of amr graph intersection and textual similarity measurement. the results for both datasets are in for the fever dataset, the best accuracy and macro f1 score are achieved when λ = 0, suggesting that the smatch precision score has minimal impact on predicting the veracity of (claim, evidence) pairs. the label-wise performance shows that am rex f ev eracc,f 1 is more effective at identifying supporting (claim, evidence) pairs but struggles with refuting instances. however, the averitec dataset exhibits different behavior, with λ = 0.9 yielding the best accuracy and λ = 0 producing the best macro f1 score. am rex av erit ecacc also manages to surpass the averitec accuracy baseline. am rex av erit ec f 1 performs comparably to the averitec baseline in recognizing refutable (claim, evidence) pairs and those with conflicting evidence. however, with greater emphasis on the smatch precision score when λ = 0.9, am rex av erit ec f 1 improves in identifying refutable (claim, evidence) pairs, albeit at the cost of performance on other label instances. through an error analysis, we identify several cases where amrex fails to accurately predict the veracity and we explore their potential causes. consider the following supporting (claim, evidence) pair from the fever dataset, claim: "wish upon was released in the 21st century.", evidence: "it is set to be released in theaters on july 14, 2017, by broad green pictures and orion pictures" (see fig. 4 for corresponding amrs in penman notation (goodman, 2020)). amrex returns the following mapping for this instance with a smatch precision score of 0.53 and a textual similarity score of 0.38. ## conclusion we implement amrex, an abstract meaning representation-based veracity prediction and explanation system for fact verification, and show its effectiveness in producing explainable justifica-amr graph mapping analysis: