Unknown Claims: Generation of Fact-Checking Training Examples from Unstructured and Structured Data (2024)

Abstract: Computational fact-checking (FC) relies on supervised models to verify claims based on given evidence, requiring a resource-intensive process to annotate large volumes of training data. We introduce Unown, a novel framework that generates training instances for FC systems automatically using both textual and tabular content. Unown selects relevant evidence and generates supporting and refuting claims with advanced negation artifacts. Designed to be flexible, Unown accommodates various strategies for evidence selection and claim generation, offering unparalleled adaptability. We comprehensively evaluate Unown on both text-only and table+text benchmarks, including Feverous, SciFact, and MMFC, a new multi-modal FC dataset. Our results prove that Unown examples are of comparable quality to expert-labeled data, even enabling models to achieve up to 5% higher accuracy. The code, data, and models are available at https://github.com/disi-unibo-nlp/unown

## introduction the spread of false information on social media threatens public trust. for example, during the covid-19 pandemic, misinformation led to vaccine hesitancy, straining public health systems and informed decision-making @xcite @xcite . computational fact-checking (fc) is a vital tool for verifying claims against diverse evidence types, including unstructured text and structured tabular data. diversity increases task complexity, requiring advanced nlp methods to cross-reference information accurately @xcite . traditional fc verification models (i.e., those making final predictions over evidence, without retrieving it) heavily rely on training samples manually annotated by experts, who meticulously review and pair claims with corresponding evidence, and intentionally modify claims to create refuting examples. unfortunately, this process is labor-intensive and time-consuming, which significantly hinders the scalability of fc efforts in adapting to evolving misinformation scenarios @xcite . recent studies have attempted to mitigate these challenges by automating the generation of training examples using question-answering (qa) and entity replacement (er) algorithms @xcite . yet, as shown in the main findings of our study are as follows: ## related work computational fc has been an active area of research for decades @xcite . recently, the rise of llms has advanced the development of fc pipelines @xcite , but their effectiveness is still inferior to human experts @xcite . specialized models are currently the most effective approach @xcite , despite they require large labeled datasets for training. existing methods for automatically generating fc training examples have been approached through both unsupervised and supervised techniques. unsupervised solutions, typically employed in the absence of labeled data, leverage plms to create textual claims from a given text, e.g., by using template prompts @xcite . supervised approaches rely on specific resources, e.g., an annotated taxonomy to train an lstm model for sentence generation @xcite . several works have investigated the generation of claims from textual evidence (see research on generating claims specifically from tabular data remains limited. while some stud-ies have explored template-based methods @xcite , @xcite demonstrated improved results by producing claims based on human-provided examples. artificial text passages have recently demonstrated greater effectiveness than human-written ones for reasoning-demanding qa @xcite , but fc tasks have not yet been studied. to the best of our knowledge, no existing work has addressed the generation of fc training examples from structured and unstructured data as input. ## problem formulation let d represent a semi-structured document (e.g., a wikipedia page) containing n sentences and m tables. we define evidence @xmath0, e t } as a nonempty subset of sentences e @xmath1, . . . , s |es|<n } and, optionally, cell values e @xmath2, . . . , c |et|<p } extracted from a table within d, where p is the total number of cells. a supervised fc model f evaluates whether a textual claim c is supported or contradicted by the given evidence e. specifically, f takes as input a data @xmath3, @xmath4, refutes}. 5 consequently, our goal is to automatically generate labeled examples @xmath5, c, @xmath6. ## method we introduce unown (figure 3 ), a novel framework to automate the production of fc training data. in a first step, e is created from the input d (evidence selection). then, e is used to generate supporting or refuting claims (claim generation). ## experimental setup our focus is on evaluating the veracity component of the fc process during test time, where models are provided with gold evidence alongside the claim for verification. to achieve this, we address the following research questions: q1 are our generated artificial examples effective for training fc models? q2 which evidence selection strategy yields the best performance? q3 what method is recommended for generating refuting claims? q4 to what extent does the efficacy of synthetic examples generalize across various domains? q5 how many internal dataset-specific samples are necessary for few-shot learning to bootstrap the downstream fc model successfully? datasets. in warm-start scenarios, we use a subset of 10k positive and 10k negative human examples from fever @xcite , a collection of claim-evidence pairs based on wikipedia. as the leading fc benchmark, we take fever-ous @xcite , an extension of fever with more complex claims enriched with tabular evidence (with no overlap between the two corpora). since the original test set is private and lacks gold labels and evidence for the claims, we used the provided validation set as our test set for evaluation. we then divided this set into two subsets: one containing claims based solely on textual evidence, and another containing claims that require both textual and tabular evidence (we exclude claims relying only on tables). to assess generality, we include scifact @xcite , a dataset of expert-written claims paired with evidence from scientific papers abstracts. for the same rationale applied to feverous, we used the original validation set as our test set. finally, we release mmfc, a new multi-modal fc corpus. mechanically, we sample 2000 instances from multimodalqa @xcite , a qa metrics. we assess fc predictions using accuracy and f1 scores ([0, 1]; higher is better), distinguishing between supports and refutes labels. we validate models on the test sets after training with artificial and human examples. we finally evaluate the logical relationship between each evidence-claim pair with a deberta cross-encoder @xcite pretrained on natural language inference (nli) tasks to classify pairs as entailment, contradiction, or neutral. claim generation models. as slm, we use models built on bart @xcite . for supporting claims, we employ the large version (400m parameters). for refuting claims, we utilize two variants: bart-large and bartneg @xcite , a specialized bart-base model (140m parameters) trained on parallel and opposing claims from the wikifactcheck dataset @xcite . 7 as llm, we operate with llama-2-7b @xcite , opting for qlora @xcite al., cold start warm start bart + er bart bartneg bart+bartneg bart bartneg bart+bartneg 0 10 100 10k 60 65 70 75 80 85 90 95 100 94.5 76.4 79.8 69 81.3 90.2 63.6 77.4 85.2 91.6 86.7 88.7 89.8 90 89.6 85.3 91.5 91.5 88.7 85.2 92.3 92.4 77.3 73.9 89 89.8 # training samples accuracy (%, ↑) 2023) adapter fine-tuning (the prompt template is provided in the appendix). we stress that refuting claim generation can be obtained by: (i) running these models directly on e; (ii) applying these models to the claim returned by a supporting model. training is done independently for the two claim types; details are reported in the appendix. ## conclusion we introduced unown, a domain-agnostic framework to automatically generate training examples for fact-checking systems, bypassing the costly task of manually annotating large volumes of data. un-own fits both structured and unstructured data to compile textual claims that support or refute the evidence provided. it also accommodates several solutions for evidence selection and claim generation to adapt to different scenarios. we evaluated our framework using three datasets that deal with general-domain and scientific contexts. the results indicate that our synthetic examples exhibit a quality comparable to that of expert-labeled data, showing the practicality and efficacy of our framework. quantitative and human evaluation also register that our refuting examples have high variety, comparable to human-generated ones.