Leveraging fine-tuned Large Language Models with L o RA for Effective Claim, Claimer, and Claim Object Detection (2024)

Abstract: Misinformation and disinformation phenomena existed long before the advent of digital technologies. The exponential use of social media platforms, whose information feeds have created the conditions for many to many communication and instant amplification of the news has accelerated the diffusion of inaccurate and misleading information. As a result, the identification of claims have emerged as a pivotal technology for combating the influence of misinformation and disinformation within news media. Most existing work has concentrated on claim analysis at the sentence level, neglecting the crucial exploration of supplementary attributes such as the claimer and the claim object of the claim or confining it by limiting its scope to a predefined list of topics. Furthermore, previous research has been mostly centered around political debates, Wikipedia articles, and COVID-19 related content. By leveraging the advanced capabilities of Large Language Models (LLMs) in Natural Language Understanding (NLU) and text generation, we propose a novel architecture utilizing LLMs finetuned with LoRA to transform the claim, claimer and claim object detection task into a Question Answering (QA) setting. We evaluate our approach in a dataset of 867 scientific news articles of 3 domains (Health, Climate Change, Nutrition) (HCN), which are human annotated with the major claim, the claimer and the object of the major claim. We also evaluate our proposed model in the benchmark dataset of NEWSCLAIMS. Experimental and qualitative results showcase the effectiveness of the proposed approach. We make our dataset publicly available to encourage further research.

## introduction in the information era where data are abundant and constantly flowing across the internet, there is a high need for developing tools that evaluate the veracity of claims. claim detection refers to the @xcite in a new study, a team from japan presents a novel method of synthesizing a promising membrane on a novel silica substrate, called si-cha @xcite now, in a new study published in membranes, scientists from japan have developed a new method to synthesize a pure si-cha membrane. @xcite the key to this achievement, as lead scientist dr. mikihiro nomura from shibaura institute of technology (sit) explains, is to use a porous silica substrate instead of the conventional alumina substrate to grow the crystal. ## related work the initial step in automated fact-checking is claim detection, where claims are identified for further verification @xcite . typically, this detection process relies on the concept of checkworthiness. previous works defined check-worthy claims as those that the general public would find interesting to know the truth about @xcite . @xcite reframed claim detection as the determination of whether a claim presents an assertive statement about the world that can be fact-checked. they focused on whether the claim is verifiable using easily accessible evidence. claims that rely on personal experiences or opinions are deemed uncheckable. in the past, political debates were the main field of interest for claim detection. claimbuster @xcite , an end-to-end system for fact-checking, is trained on human-annotated sentences sourced from previous general election debates. claimbuster incorporates "claim spotter", a component designed to determine the probability of a sentence containing claims that could be verified. @xcite introduced an openly available dataset derived from the 2016 us presidential and vice-presidential debates. they applied a context modeling approach on both their dataset and claimbuster @xcite and stateof-the-art performance was achieved. most recently, @xcite tried to detect check-worthy claims from question-hour debates of the indian parliament, tweets posted by politicians, and prime minister statements. several studies have also focused on claims extracted from wikipedia articles. @xcite presented a manually created argumentative dataset sourced from wikipedia articles. the dataset comprises 2,683 argument elements that cover 33 controversial topics in claim-evidence pairs. @xcite utilized the above dataset to introduce the task of context-dependent claim (cdc) detection. they define cdc as "a general, concise statement that directly supports or contests the given topic". most recently, given the widespread impact of the covid-19 pandemic, numerous works have emerged that specifically address claims related to covid-19. @xcite introduced news-claims, a manually created dataset for claim detection consisting of 143 covid-19 related news articles with 889 annotated claims. each claim has additional attributes, which are the claimer, the claim object, and the claim stance. gangi reddy et al. (2022) further used newsclaims benchmark to evaluate on it a pre-trained question-answering (qa) system. both works are thematically restricted, as they used a set of predefined topics regarding covid-19. similarly, covid-19 claim radar @xcite is a system that integrates claim extraction and knowledge extraction to provide users with a structured and comprehensive understanding of claims associated with the covid-19 pandemic. clef checkthat! is a shared task that began in 2018 and focuses on the automatic identification and verification of claims made on twitter and political debates. most recently, at clef check-that! 2022 @xcite , where the focus was on tweets regarding covid-19 and politics, transformer models such as bert @xcite and gpt-3 @xcite achieved the best performance. lately, llms have also been leveraged for claim detection. @xcite created the self-checker, a framework of plug-and-play llms modules for automatically fact-checking, which was evaluated on fever @xcite and wice @xcite datasets. @xcite also utilized llms and employed them to automatically generate claims for data augmentation. overall, up to now works on claim detection have focused on unstructured texts (e.g., political debates, tweets), while works on structured texts such as wikipedia texts and news articles are domain specific, for example on the covid-19 pandemic. additionally, most of those approaches and existing automated claim detection systems prefer to define a factual and check-worthy claim, i.e. a segment of text containing measurable data, numbers or percentages, survey results, and relevant metrics. those approaches may work well on unstructured text (tweets, speeches, etc.), but are weak when tackling structured text such as news articles. furthermore, a non-specialist reader when confronted with texts presenting scientific data, research numerical results, etc. can benefit little from any verdict that a system gives on a particular claim containing such information. contrary to prior work, we locate the claim that summarises the main idea of a science-related news article, that is, the "major claim" of the text, since this is what a reader most easily identifies and is influenced by. the need for such an approach is high, as original findings from science publications may be distorted when reported to the public. this can lead to misinformation spread and, consequently, those altered versions of the original findings in the reporting may not be as accurate, possibly due to the different writing purposes between scientists and journalists @xcite . ## datasets in this section, we describe the datasets used in our methodology and experiments. we start with our hcn dataset and provide details on how we built it, and then move to the newsclaims dataset, putting emphasis on its different sub-tasks and how these correlate with our task. ## methodology generative ai is undergoing impressive growth with llms leading the way. llms are intricate models consisting of billions of parameters, trained on extensive collections of text and have demonstrated exceptional effectiveness across a broad spectrum of text-related assignments. it is worth mentioning that these models still remain language models, trained on the text completion task (i.e. predict the next token), enabling them to learn and generalize from extensive and diverse training data. these models however must be fine-tuned in specific tasks in order to be effective and to better adapt to specific domains or types of text that were not well represented in their original training data. to that end, we employ instruction fine-tuning to tune the model using examples of the target task. instruction fine-tuning involves utilizing a collection of labeled examples represented as promptresponse pairs to enhance the pre-trained model's ability to accurately predict the response based on the given prompt. ## experiments we investigate the effectiveness of our proposed approach in the two datasets, namely hcn and news-claims. in the appendix section a.3 we outline the details of our implementation, the hyperparameter tuning performed in the hcn dataset and the optimal sets of parameters. in the next sections, we discuss the evaluation setting, the baselines used in the comparative analysis and the results of our experiments in the three sub-tasks outlined in this paper.