C laim L ens: Automated, Explainable Fact-Checking on Voting Claims Using Frame-Semantics (2024)

Abstract: We present ClaimLens, an automated fact-checking system focused on voting-related factual claims. Existing fact-checking solutions often lack transparency, making it difficult for users to trust and understand the reasoning behind the outcomes. In this work, we address the critical need for transparent and explainable automated fact-checking solutions. We propose a novel approach that leverages frame-semantic parsing to provide structured and interpretable fact verification. By focusing on voting-related claims, we can utilize publicly available voting records from official United States congressional sources and the established Vote semantic frame to extract relevant information from claims. Furthermore, we propose novel data augmentation techniques for frame-semantic parsing, a task known to lack robust annotated data, which leads to a +9.5% macro F1 score on frame element identification over our baseline.

## introduction the proliferation of misinformation and disinformation in today's digital landscape has highlighted the urgent need for effective and efficient fact-checking solutions. manual fact-checking is time consuming and is often too slow to stop the early spread of misinformation. automated fact-checking methods have emerged as a promising approach to combating the spread of false information. early approaches formulated queries for databases @xcite and knowledge graphs @xcite ; however, with the strength of large language models, most of the existing systems rely on machine learning models @xcite @xcite which suffer from a critical limitation: a lack of transparency and explainability. to alleviate this problem, some systems have incorporated an explanation element @xcite which generates explanations for their predictions. but these posthoc explanations can result in the model justifying incorrect predictions or hallucinating facts to justify a correct prediction. the opacity of these models can lead to a trust deficit, making it difficult for users-particularly journalists, researchers, and policymakers-to understand the reasoning behind the fact-checking outcomes. this limitation is particularly concerning in high-stakes domains, such as journalism, healthcare, and finance, where the credibility of fact-checking results is paramount. recent works towards automating fact-checking are primarily focused on fake news/misinformation detection @xcite and fact verification @xcite . fake news detection is generally defined as the identification of news containing nonfactual statements, often with malicious intention to mislead the public @xcite . this is typically done by building models which look at a combination of features such as linguistic cues, user statistics, and news sources, without necessarily determining the truthfulness of the statements. fact verification is the process of verifying whether a particular claim is true or false given a piece of evidence @xcite . fact verification methods assume that the piece of evidence is given. however, this is not always the case for real-world claims which are often self-contained and lack supporting evidence. thus, to fact-check a claim, it is necessary to couple fact verification with an effective evidence retrieval method. in this work, we explore the use of framesemantics @xcite )-a structured method of extracting important segments from a sentence-in evidence retrieval and fact verification, in order to produce an end-to-end automated, explainable fact-checking system. framesemantic parsing @xcite is the process of automatically identifying semantic frames (frame identification) and frame elements (argument identification) within text. semantic frames are structured events, concepts, or scenarios containing frame elements (fes) which describe different roles or entities related to the frame. semantic frames provide a structured framework for performing and explaining natural language processing tasks and has been previously used for knowledge extraction @xcite , question answering @xcite , and event detection @xcite . this study focuses on voting-related factual claims, as it is a domain where a large amount of structured, trustworthy data are available in the form of voting records. to do this, we utilize the vote frame defined in @xcite . given a particular voting-related claim, we subject it to argument identification to extract the agent, issue, and position fes, which correspond to the voter, what they are voting on, and what their position is, respectively. (an example claim with its fes can be found in figure 1 .) the truthfulness of the claim can be verified by corroborating or refuting the extracted fes using a database of public voting records, specifically the united states congressional voting records in our system. the database contains a large number of bills and their descriptions, as well as many congress members and their voting records on the bills. the extracted agent fes and issue fes are matched with the congress members and bills in the database. while finding the corresponding congress member given an agent fe is straightforward using a simple keyword search, matching an issue fe with bills is considerably more challenging. in this work, we analyze several text search approaches for matching issue fes to their respective bills. to evaluate these search methods, we collected a new dataset (details in section 3.2) of voting-related claims from politi-fact fact-checks and their corresponding bills from the content of the fact-checks. to perform the frame-semantic parsing, we use the system described in @xcite for frame identification and build on the work in @xcite for argument identification. to overcome the limited data for the vote frame in @xcite , we developed two strategies for data augmentation, including fe interleaving and fe permutation (detailed in sections 3.3.1 and 3.3.2, respectively). fe interleaving takes two annotated sentences with the same frame and swaps combinations of fes between the two sentences to create new ones. fe permutation uses a single annotated sentence to create new sentences by reordering the fes in the original sentence. while voting-related claims is a limited scope, this work can be applied using any frame, given there is sufficient data available, and we plan to expand this work in the future into a few other feasible domains, e.g., verifying claims related to oecd countries using public datasets on their gdps, crime rates, education rankings, and so on. we summarize our contributions below. ## demonstration in this section, we demonstrate the functionality of our system using the fact-checked claim, "marsha blackburn voted against a military pay raise," as cited in @xcite . the demonstration showcases the key components of our system, from claim analysis to evidence retrieval and alignment. first, the system analyzes the semantic structure of the claim to identify the key elements, specifically the agent (marsha blackburn) and the issue (military pay raise), as illustrated in figure 1 . the agent lookup process involves retrieving information about the relevant congress member, including their unique identifier, an image, and a brief biography from wikipedia, as shown in figure 4 . next, the system searches for legislative bills related to the identified issue. it retrieves the voting records of the specified congress member on these bills. for each relevant bill, the system computes the alignment between the claim and the vote, utilizing the methodology discussed in section 4.3. figure 3 shows one of the resulting bills from our bill search model including the bill title/identifier, a summary of the bill, the congress member's vote on the bill, and the alignment of the claim to the bill. in this example, marsha blackburn voted for the department of defense appropriations act of 2016, which specifically includes provisions for military personnel. for this bill, our claim alignment model determined that this vote refutes the claim because "the bill summary indicates that the department of defense appropriations act, 2016 provides appropriations for military personnel, which would generally include funding for military pay raises. marsha blackburn's vote was 'aye', meaning she voted in favor of this bill. therefore, the claim that 'marsha blackburn voted against military pay raise' is incorrect as per this voting record." ## conclusion and future work in this work we introduced claimlens, the first system which utilizes frame-semantic parsing for explainable, automated fact-checking. additionally, we outlined important challenges and detailed our methods to solve them, namely on semantic bill search and vote-claim alignment. we also constructed and released our us congress database and our annotated bill matching evaluation set. furthermore, we introduced and evaluated two novel data augmentation techniques for frame-semantic parsing which significantly improve the model's performance. these achievements lay the foundation for explainable, automated fact-checking with frame-semantics. in a future study, we aim to expand the scope of the fact-checking capabilities using other frames in @xcite . one such example is the occupy_rank frame which is about items occupying a certain rank within a hierarchy. for example, consider the claim "the u.s. has the 6th highest poverty rate among oecd countries." using this frame, we could extract "the u.s." as the item, "6th" as the rank, "poverty rate" as the dimension, and "oecd countries" as the compari-son_set. then, a query could be formed to deter-mine whether the claim is true. we also plan to investigate alternatives to llms for vote-claim alignment due to speed demands for our system. specifically, we would like to represent this as a textual entailment task to utilize the vast research available on textual entailment methods. finally, we would also like to apply our data augmentation techniques to the original framenet dataset to evaluate of the generalizability of our augmentation techniques.