AF a CTA : Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators (2024)

Abstract: With the rise of generative AI, automated fact-checking methods to combat misinformation are becoming more and more important. However, factual claim detection, the first step in a fact-checking pipeline, suffers from two key issues that limit its scalability and generalizability: (1) inconsistency in definitions of the task and what a claim is, and (2) the high cost of manual annotation. To address (1), we review the definitions in related work and propose a unifying definition of factual claims that focuses on verifiability. To address (2), we introduce AFaCTA (Automatic Factual Claim deTection Annotator), a novel framework that assists in the annotation of factual claims with the help of large language models (LLMs). AFaCTA calibrates its annotation confidence with consistency along three predefined reasoning paths. Extensive evaluation and experiments in the domain of political speech reveal that AFaCTA can efficiently assist experts in annotating factual claims and training high-quality classifiers, and can work with or without expert supervision. Our analyses also result in PoliClaim, a comprehensive claim detection dataset spanning diverse political topics.

## introduction the explosion of mis-and disinformation is a growing public concern, with misinformation being widely shared @xcite . manual factchecking is an important counter-measure to misinformation @xcite . however, fact-checking is a time-consuming and expensive endeavor, and computational remedies are required @xcite . a first step to identify mis-and disinformation consists of factual claim detection, which filters out the claims with factual assertions that need checking @xcite @xcite . considering the sheer amount of daily online content and llms' generative capability, we argue that a valid factual claim detection system should be efficient and easily deployable to monitor misinformation consistently. therefore, we need a way to produce high-quality resources to build transparent, accurate and fair models to automatically detect such claims. however, there are two major challenges in the data collection process. discrepancies in task and claim definitions. by now, arguably, several different claim definitions exist, which confuse practitioners. what is a claim is unclear, leading to various claim detection tasks, e.g., in automated fact-checking and argument mining. for example, @xcite dismiss all opinions from factual claims, but @xcite includes "opinions with social impact" as factual claims. many studies @xcite aim at detecting "check-worthy" claims while @xcite argues the definition of "check-worthiness" is highly subjective and political. such variances reflect a lack of clarity in conceptualizing critical distinctions, such as the overlap between opinions and verifiable facts (refer to manual annotations are expensive. all existing datasets are manually annotated, which is timeconsuming and expensive. thus, most existing resources are inevitably restricted to certain topics for which it is feasible to annotate claims manually. such examples include presidential debates (has-type examples and explanations ## afacta this section introduces afacta for assisting factual claim annotation. afacta consists of three prompting steps and an aggregation step (illustrated in figure 1 ), inspired by @xcite and our claim definitions. the prompts can be found in appendix c. ## policlaim dataset we obtain a large political speech data from @xcite , which mainly consists of state of the state (sots) speeches (already cleaned and split into sentences). these speeches are governors' major public addresses of the year, thus in-cluding meaningful political topics. we randomly sample two speeches from each year, from 1998 to 2021, as training data and four speeches from 2022 as test data. 2 this design has two considerations: (1) we aim to replicate the real-world scenario where models are trained on previous claims (e.g., from 1998 to 2021) and used to predict future claims on potentially unseen topics (e.g., in 2022). (2) the test set will be used to evaluate the annotation performance of afacta, and the 2022 speeches are likely unseen by june llm checkpoints we use to better replicate the future-claimdetection scenario. the policlaim test set (policlaim test ) was annotated by two human experts foot_3 , who had no access to afacta's output when annotating. the experts achieved a substantial cohen's kappa of 0.69 in independent annotation before the discussion. then, they had meetings to resolve disagreements and develop gold labels. disagreements were mainly caused by ambiguous verifiability, see appendix a for disagreement resolving. our annotation guideline, an instantiation of our factual claim definition, can be found in appendix b. to test afacta's annotation performance on different domains, we re-annotate the development set of checkthat!-2021 @xcite , which originally contained check-worthiness labels of covid-19 tweets, following the same annotation process (cohen's kappa 0.58). due to budget limitations, our explorations and annotations mainly focused on the domain of political speech. we leave the extensive study on the social media domain (and other potential domains for factual claim detection) to future work. after verifying the performance of afacta using the test sets (see more in section 5.1), we annotated the training set with the tool's assistance, imitating its expected use case of assisting annotation. the perfectly consistent samples were labeled directly with gpt-4 afacta, while the inconsistent samples were left for human annotation. we randomly sampled 8 speeches and manually relabeled the inconsistent annotations from afacta, leading to policlaim gold where all annotations are labeled with perfect consistency or human supervision. the perfectly consistent samples in the rest of the speeches fall into policlaim silver while the inconsistent samples fall into policlaim bronze . the statistics of datasets can be found in ## experiments since afacta is an llm-agnostic prompting framework, we test both gpt-3.5 @xcite and gpt-4 (openai, 2023) as the backbone llm. we also test open-sourced llms which does not work well due to high position bias in step 3 (see appendix f). detailed settings are in appendix g to ensure reproducibility. ## related work claim detection: the term "claim detection" has different definitions in various research fields @xcite . even inside the field of factchecking, its exact definition depends on the domain @xcite or task objective @xcite @xcite and is somewhat arbitrary. in this work, we propose a definition focusing on one important dimension of factual claims -verifiability, to minimize the conceptual uncertainty. another important dimension of factual claims is check-worthiness @xcite @xcite , whose definition is more arbitrary @xcite . automatic annotation: automatic data annotation using llm is both promising @xcite and necessary @xcite . early work observes that llms' annotation performance highly depends on tasks: llms outperform human annotators on some tasks @xcite @xcite but fails to achieve human-level performance on others @xcite . therefore, we argue that a detailed task-specific study about llm annotation reliability is essential. @xcite recommend evaluating llms' annotation against a small subset that is not in the llms' training corpus and annotated by subject matter experts. we follow these suggestions in this work. concurrent studies also explore self-consistency @xcite and cot @xcite to improve the performance and reliability of llm annotation. however, they do not compare predefined reasoning paths with automatically sampled cots. ## conclusion we propose afacta, which leverages llms to assist in the annotation of factual claim detection. it ensures reliability by calibrating annotation quality through consistency. afacta's consistent annotation proves effective for training and data augmentation even without human supervision.