A Thesis Proposal C laim I nspector Framework: A Hybrid Approach to Data Annotation using Fact-Checked Claims and LLM s (2024)

Abstract: This thesis explores the challenges and limitations encountered in automated fact-checking processes, with a specific emphasis on data annotation in the context of misinformation. Despite the widespread presence of misinformation in multiple formats and across various channels, current efforts concentrate narrowly on textual claims sourced mainly from Twitter, resulting in datasets with considerably limited scope. Furthermore, the absence of automated control measures, coupled with the reliance on human annotation, which is very limited, increases the risk of noisy data within these datasets. This thesis proposal examines the existing methods, elucidates their limitations and explores the potential integration of claim detection subtasks and Large Language Models to mitigate these issues. It introduces ClaimInspector, a novel framework designed for a systemic collection of multimodal data from the internet. By implementing this framework, this thesis will propose a dataset comprising fact-checks alongside the corresponding claims made by politicians. Overall, this thesis aims to enhance the accuracy and efficiency of annotation processes, thereby contributing to automated fact-checking efforts.

## introduction the initial step in researching misinformation necessitates a set of criteria to determine the accuracy of a claim. due to the impracticality of manually scrutinising each piece of information, researchers often rely on the evaluations of factchecking organisations. they construct datasets that consist of claims that have previously been fact-checked. however, these datasets also come with a set of limitations. although a wealth of fact-checking resources exists to document the infiltration of misinformation across various channels, including political ads, politicians' websites and newspapers, the majority of current efforts concentrate on analysing textual claims from a single source, with twitter being the predominant platform for claim collection. in addition, due to the methods applied in claim matching and the lack of additional controls, the datasets generated often carry a high risk of containing a considerable amount of noisy data. efforts have been made to mitigate this issue through human annotation; however, limited resources allow such annotation to be performed on only a limited portion of the data @xcite @xcite . as a result, all these limitations may pose a risk of reduced efficacy in detecting misinformation, since claim detection and fake news detection models may be trained on this limited -and potentially noisy -subset of data. therefore, this thesis proposal is centred on addressing the limitations of this process. informed by these challenges, the main objective of this thesis is to answer these questions: this thesis considers the multimodality of misinformation across various channels. it aims to refine the matching process by drawing on automated fact-checking literature and seeks to establish a more efficient annotation process by incorporating llms into the annotation workflow. by improving this process, this study seeks to not only contribute to the automated fact-checking process, but also to provide support to fact- ## related work in this section, i review the literature on data annotation and claim detection and discuss the limitations. researchers divide the fact-checking pipeline into four main subtasks: (1) the assessment of checkworthiness, (2) the detection of previously fact-checked claims, (3) the retrieval of evidence and (4) the verification of the factuality of the claim @xcite . data annotation, while not listed among these subtasks, can be considered a preliminary task (0). this foundational step is crucial, as it involves labelling data, which supports both the preparation of data for the entire fact-checking pipeline and the training of algorithms. claim detection, another integral part of this sequence, is closely linked to the assessment of checkworthiness and the detection of previously fact-checked claims. ## proposed method the proposed method consists of two main stages. the first stage, outlined in section 3.1, involves extracting the source link of a claim from a factchecking article and verifying its relevance to the fact-check. the second stage concentrates on broadening the dataset's scope by retrieving relevant news articles or video content associated with a verified claim, as elaborated on section 3.2. subsequently, section 3.3 introduces the claiminspector framework, providing a summary of the overall process. ## implementation: a case study on claims made by politicians while applicable for collecting and annotating data across diverse topics, this thesis will employ the framework to identify claims made by united states (us) politicians. i will scrape both politifact and snopes, which are ifcn-accredited factchecking organisations. gathering data from two fact-checking organisations will give us a more balanced and diverse view of fact-checked claims. politifact primarily concentrates on scrutinising claims associated with politicians, and its sample of politicians is representative of the population of ## experiments on the claiminspector: politicians edition in order to assess the claiminspector, i will conduct two sets of experiments. first, i will perform a claim detection task using bm25 and bert-based models. as the evaluation measure, i will calculate mean reciprocal rank, mean average precision and mean average precision at k for @xmath0, 3, 5, 10, 20, 30}. the results from the clef-2022 checkthat! lab task 2b will serve as the baseline for this experiment @xcite . the second set will focus on the annotation results of pre-trained llms. due to the significant computational resources required for fine-tuning llms, this process falls outside the scope of this thesis. i plan to conduct experiments with zero-shot prompting @xcite and few-shot prompting @xcite . the performance of llms can also be significantly influenced by the prompts given. as such, i will experiment with several elements of the prompts, including the choice of words and the structure of sentences. to evaluate the efficacy of the llms, i will employ a range of performance indicators, including macrolevel precision, recall and accuracy. the outputs from the models will be compared with benchmark annotations provided by human annotators. ## conclusion this thesis proposal offered an analysis of the current annotation methods and suggested enhancements through additional controls, including similarity measures and llm-guided annotation. it advocated for broadening the range and type of claim sources beyond mere textual content and social media. additionally, the proposal underscored the need to identify content akin to sources cited in fact-checking articles, acknowledging the extensive reach of misinformation beyond the sources typically listed in fact-checking articles. to systematically implement these contributions, this thesis proposal introduced the claiminspector framework, a novel hybrid approach to data annotation. the proposal outlined the plans for applying this framework by creating a dataset called claiminspector: politicians edition. through this dual focus on methodology improvement and dataset creation, the research intends to equip researchers and factcheckers with reliable tools.