Task-Oriented Automatic Fact-Checking with Frame-Semantics (2025)

Abstract: We propose a novel paradigm for automatic fact-checking that leverages frame semantics to enhance the structured understanding of claims and guide the process of fact-checking them. To support this, we introduce a pilot dataset of real-world claims extracted from PolitiFact, specifically annotated for large-scale structured data. This dataset underpins two case studies: the first investigates voting-related claims using the Vote semantic frame, while the second explores various semantic frames based on data sources from the Organisation for Economic Co-operation and Development (OECD). Our findings demonstrate the effectiveness of frame semantics in improving evidence retrieval and explainability for fact-checking. Finally, we conducted a survey of frames evoked in fact-checked claims, identifying high-impact frames to guide future work in this direction.

## introduction the proliferation of misinformation poses a critical challenge to the modern information ecosystem, threatening informed decision-making and public trust. addressing this challenge requires scalable and reliable methods to verify the accuracy of claims-particularly novel ones that human fact-checkers may not have previously encountered. automatic fact-checking has thus emerged as a crucial research area, aiming to reduce the manual burden of verifying claims by automating key steps in the fact-checking pipeline. much of the previous work on automatic factchecking has focused on unstructured data, such as fact-checks from trustworthy sources, to perform claim matching @xcite and use large language models to produce fact-check verdicts and explanations @xcite @xcite . structured tabular data from wikipedia @xcite and scientific documents @xcite have also been utilized to vet synthetic claims extracted from wikipedia @xcite and real-world claims @xcite . however, these studies only consider data which have already been processed and prepared for easy consumption by readers. this reliance on highlycurated data hinders their ability to fact-check novel claims which do not have such clean evidence readily available. to bridge this gap our work proposes a novel pilot dataset of real-world factual claims related to two high-volume structured databases on u.s. congressional voting records and country statistics. these databases significantly differ from past works in both volume and detail of their tables. for example, wikipedia tables consist of an average of 13 rows @xcite while our database of country statistics has an average of 596,000 rows across over 400 tables. this scale of data is incompatible with most previous methods, which often rely on large language models (llms), as the tables themselves are too large to be analyzed due to context length limitations, even for those solutions designed for large tables @xcite . fact-checking novel claims requires fine-grained methodology to understand the claim and to map important parts of the claim to their relevant data. furthermore, transparency and explainability are critical in fact-checking. we identify frame semantics @xcite as a viable solution which addresses each of these points. frame semantics is a linguistic framework that explores how meaning is encoded through structured represen-tations called frames. these frames capture the essential elements and relationships of the situations described by claims, supporting explainable understanding and extraction of key components from the claims. the existence of specific frames in a claim also enables fine-grained task-specific handling of the claim for fact-checking. in our paradigm, we use the frames evoked, and manuallyselected frame elements, in a given claim to identify relevant database tables for automatically factchecking it. our proposed pilot dataset serves as the foundation for two case studies that demonstrate how frame semantics can be used to guide specific factchecking processes and enhance the explainability of automatic fact-checking. in the first case study, we focus on the vote frame @xcite , which models how an agent (e.g., a legislator) interacts with an issue (e.g., a bill) through the act of voting and optionally takes a position (for or against) on the issue. these issue and agent elements are then used to implement voting-specific fact-checking methods. the second case study follows similar principles on a much more diverse set of frames applied to a high-volume, highly diverse structured database produced from hundreds of datasets collected from the organisation for economic co-operation and development (oecd). to evaluate these case studies, we collected 79 and 68 real-world claims for the voting records and oecd statistics, respectively. each claim is manually annotated for its particular frames and factcheck verdict retrieved from politifact. our case studies demonstrate that frame semantics offers a valuable framework for task-specific fact-checking. frame-semantics can also offer additional performance benefits for particular tasks. for example, on vote and oecd-related claims, retrieving relevant evidence by querying with extracted taskspecific frame elements outperformed using the entire claim by 14% and 11%, respectively. the performance of particular retrieval methodologies also significantly varied between the two case studies, indicating a need to select different approaches for each specific task to achieve optimal performance. furthermore, instructing large language models (llms) to fact-check claims benefits from different prompts depending on the type of claim. these factors further showcase the benefit of a taskspecific approach to automatic fact-checking compared to a suboptimal generalized approach. finally, to understand which frames are most commonly evoked in factual claims, we surveyed claims fact-checked by politifact and identified a heavy skew toward a few highly frequent frames. notably, the frames used in our case studies are among the most frequently evoked frames in poli-tifact fact-checks. this alignment highlights the practical relevance of our approach, as it directly addresses the semantic structures most commonly encountered in professional fact-checking. to summarize, our contributions are as follows: • we proposed a novel paradigm for task-oriented automatic fact-checking using frame-semantics. automatic fact-checking. recent advances in automatic fact-checking have been largely driven by the integration of large language models (llms) and retrieval-augmented generation (rag) pipelines. @xcite proposed a unified framework for llm-based systems which utilizes an internal mechanism to determine which of a selection of llm-base models should be used to factcheck a particular claim. ragar @xcite enhances fact-checking by using multimodal inputs and iterative reasoning. factllama (cheung and lam, 2023a) combines a pre-trained llama model with external evidence retrieval to fact-check claims. llm-augmenter @xcite integrates external knowledge sources and provides feedback to improve model accuracy. @xcite mitigates misinformation generated by rag pipelines via re-ranking retrieved documents according to a credibility score. for structured data, past work has studied using fine-tuning methods @xcite @xcite and llm-based methods @xcite @xcite for fact verification. dater @xcite is a fact-verification system which simplifies the factchecking process by decomposing claims into subquestions using llms. it is the best-performing model on the tabfact dataset @xcite . our work does not primarily focus on fact verification; rather, our main emphasis is on evidence retrieval. nevertheless, we employ llm-based methods for the fact-verification step, as they offer the practical advantage of not requiring model retraining when new data becomes available. applications of frame semantics. madabushi (2024) compares search methods for finding textual entailment facts in rag-based systems, including a frame-based method. the authors found that their frame-based approach significantly improved recall@k compared to other search-based methods. additionally, many previous studies have used frame semantics, or similar semantic parsing methods, in creating task-oriented systems in other nlp domains @xcite @xcite . ## fact-checking paradigm we break down the task of fact-checking into three key steps: claim understanding, evidence retrieval, and fact verification. figure 1 shows an example of how our system processes the statement "japan has the 2nd highest life expectancy." ## fact-checking case studies in this section we provide two case studies which utilize our proposed paradigm. ## conclusion in this work, we introduced a novel paradigm for task-oriented automatic fact-checking by leveraging frame semantics to enhance structured under-standing and evidence retrieval. through the development of a pilot dataset composed of real-world claims and two focused case studies, we demonstrated how frame-semantic parsing can guide the fact-checking process with structured data, improving retrieval performance and explainability. our findings suggest that semantic frames offer a meaningful way to bridge the gap between complex claims and high-volume structured datasets. the empirical results highlight the advantages of using frame elements for evidence retrieval, with improvements in recall across both voting-related and oecd-based fact-checking tasks. by aligning claims with relevant structured data, our approach not only enhances accuracy but also provides a more interpretable verification process. furthermore, our survey of fact-checking frames reveals promising avenues for future research, particularly in expanding the coverage of high-impact frames and refining claim-evidence alignment techniques. while our system is limited by the availability of trustworthy data, our findings highlight a critical research gap left by approaches that focus on smaller tabular data. this work showcases the use of highvolume structured databases for automatically verifying novel claims. expanding beyond our pilot datasets to large-scale collections of claims and corresponding evidence from structured sources remains an important next step.