A Semantics-Aware Approach to Automated Claim Verification (2022)

Abstract: The influence of fake news in the perception of reality has become a mainstream topic in the last years due to the fast propagation of misleading information. In order to help in the fight against misinformation, automated solutions to fact-checking are being actively developed within the research community. In this context, the task of Automated Claim Verification is defined as assessing the truthfulness of a claim by finding evidence about its veracity. In this work we empirically demonstrate that enriching a BERT model with explicit semantic information such as Semantic Role Labelling helps to improve results in claim verification as proposed by the FEVER benchmark. Furthermore, we perform a number of explainability tests that suggest that the semantically-enriched model is better at handling complex cases, such as those including passive forms or multiple propositions.

## introduction with the rise of digital channels that disseminate all kinds of information, misinformation has become a big challenge for a healthy society @xcite . fake news has been defined as a news article or message published through media that carries false information @xcite . although this is not a new phenomenon, the current absence of control systems in social media facilitates the fast spreading of misinformation, arriving to a large number of users and greatly influencing their perception of real world events @xcite . recent work has shown that fake news spread faster in social media than factual news @xcite , which is why researchers from different fields have proposed using automated solutions to help dealing with this situation @xcite . claim verification is the task of assessing the veracity of a statement by finding evidence about the claimed facts. this work is usually done manually by fact-checkers, who use their trusted sources to label the claims as true, false or other assessments. automated claim verification, as proposed by @xcite , consists in, given a claim, finding the evidence regarding the veracity of that claim to then infer its truth-label. systems for automated claim verification have been trained both using synthetic data @xcite , and crawling datasets from fact-checking websites @xcite . these datasets have enabled the development of models for the three tasks involved in the claimverification pipeline: document retrieval @xcite , sentence retrieval @xcite , and natural language inference @xcite . in this work, we focus on the last module: natural language inference (nli). given the right pieces of evidence, a factchecking system will have to reason over all the utterances involved in order to determine if the claim can be supported, refuted, or whether there is not enough info to do so. in figure 1 , for instance, it should recognize that the rodney king riots is the same entity in the claim and in evidence 1. then, it should identify that the location of this event is los angeles county, and understand that evidence 2 confirms that this happens to be the most populous county in the usa. as illustrated in figure 1 , this reasoning process requires a deep understanding of the semantics of all the utterances involved. in this work, we propose to introduce explicit semantic knowledge in order to improve the systems for automated claim verification. we hypothesize that this information might guide the natural language inference model in claims that have complex semantics. the linguistic information we use in this work is semantic role labelling @xcite and open information extraction (openie, 2020 ) @xcite . in our experiments, these semantic structures are used as additional input to the bert contextualized word embeddings @xcite . we integrate this information using the sembert architecture presented in @xcite . the contributions of this work are the following: in the following sections, we introduce previous work on datasets, systems and semantic structures (section 2), we explain our experiments (section 3) and expose the primary results (section 4), we perform explainability tests to qualitatively assess the influence of semantic structures (section 5), and finally we draw our conclusions and future work (section 6). ## related work automated claim verification is a relatively new task, and a lot of effort have been put on how to develop datasets to train automated systems for this task. in the following subsections we introduce some of these efforts and the systems that have been developed on these datasets. we also present previous work using semantic structures. ## results going back to our hypothesis that claim complexity will be better understood by using models that include srl, we calculate the accuracy separately for claims with more (and with less) than 5 verbs. the sembert_tags1 model improves 6.5 points on complex claims over bert, while it just improves 1.5 points on simple claims. however, since fever has few complex claims (only 62), further experiments with more complex claims should be used to confirm our hypothesis. our approach is similar to the one in dream, as both integrate semantic information to improve the reasoning process. however, instead of using a graph-based approach, we use the sembert architecture to incorporate the semantic information. as observed, dream performs better than our model, suggesting that graph-based architectures might be a better representation for semantic information. finally, the highest scoring system is proofver foot_1 . furthermore, both dream and proofver rely on better evidences, as shown by the f1 in ## explainability tests while the accuracy results allow for a comparison between models, they are not enough to understand the contribution of the semantic information to the model. for this reason, we decided to perform qualitative explainability tests based on calculating saliency scores and performing adversarial attacks. ## conclusion and future work in this work we have investigated if semantic information could help to improve the reasoning process when inferring the truth label of a claim given some pieces of evidence. to this goal, we have used two different semantic parsers and the architecture of the pre-trained model sembert @xcite . for our experiments, we have used the fever dataset @xcite , which requires building a model that, given some pieces of evidence, can output if a claim is supported, refuted, or the evidence does not give enough information. we have performed several experiments on top of the sembert architecture, such as training models with different kinds of semantic information, different sets of semantic tags, and with an additional attention mechanism to represent the semantic information. in terms of label accuracy, all our experiments have outperformed the baseline, which was a bert model with no additional semantic information. our best model uses semantic role labels and a set of 10 different tags, with no additional attention mechanism. this model achieves a label accuracy of 75.37 on the development set and 72.18 on the test set, outperforming the baseline by 1.5 and 1.3 percentage points respectively. future work could include testing the impact of these semantic structures in models such as roberta @xcite or xlnet @xcite . to better understand the contribution of the semantic information, we have performed some explainability tests with our best model. these have shown that the srl knowledge might be contributing to guiding the model in semantically complex sentences that include several propositions or passive forms. to keep moving towards systems that can contribute to the work of fact-checkers, future research on claim verification should take two directions. on the one hand, there is a need to develop large datasets that are more similar to naturally-occurring claims. on the other hand, nli models for claim verification should output more explanatory justifications to their conclusions, which would make these systems more trust-worthy. in this work, we have not dealt with the task of evidence retrieval. in fever, this task is limited by the static wikipedia database that comes with the dataset. however, in real-world scenarios defining the boundaries of what is trust-worthy information is a challenge that goes beyond research in nlp and reaches the fields of journalism, politics and even philosophy. the non-static nature of what is a true fact is an additional challenge to evidence retrieval.