Automated Claim Detection for Fact-checking: A Case Study using N orwegian Pre-trained Language Models (2023)

Abstract: We investigate to what extent pre-trained language models can be used for automated claim detection for fact-checking in a low resource setting. We explore this idea by fine-tuning four Norwegian pre-trained language models to perform the binary classification task of determining if a claim should be discarded or upheld to be further processed by human fact-checkers. We conduct a set of experiments to compare the performance of the language models, and provide a simple baseline model using SVM with tf-idf features. Since we are focusing on claim detection, the recall score for the upheld class is to be emphasized over other performance measures. Our experiments indicate that the language models are superior to the baseline system in terms of F1, while the baseline model results in the highest precision. However, the two Norwegian models, NorBERT2 and NB-BERT_large, give respectively superior F1 and recall values. We argue that large language models could be successfully employed to solve the automated claim detection problem. The choice of the model depends on the desired end-goal. Moreover, our error analysis shows that language models are generally less sensitive to the changes in claim length and source than the SVM model.

## introduction with the growing concerns about misinformation, fact-checking has become an essential part of journalism. to mitigate the time and the human burden of fact-checking and to allow for more factchecked articles, automated fact-checking (afc) systems have been developed @xcite @xcite . to approach automated fact-checking, three basic tasks are defined in the pipeline: claim detection, evidence retrieval, and claim verification. claim detection refers to monitoring social media and political sources for identifying statements worth checking. the subsequent components retrieve reliable documents for debunking the detected claims and generate a verdict. several tools have been developed to automate these tasks to meet the expectations of the human fact-checkers foot_0 . according to the studies on the user needs of factcheckers, claim detection receives the highest preference among other afc tools @xcite . automated claim detection is a classification problem, where models are trained on sentences parsed from text documents and labelled by humans according to their checkworthiness @xcite . in this work, we explore how well norwegian pre-trained language models (lms) perform on the task of automated claim detection. this is, to the best of our knowledge, the first attempt at automated claim detection for norwegian using lms. fine-tuning lms for the task of automated claim detection is not novel @xcite @xcite . however, this has never been done on norwegian, and we believe that our insights into which errors these models do compared to simple baselines is a valuable contribution. our research questions are: to address these questions, we first fine-tune each model on a small dataset from a norwegian non-profit fact-checking organization, comprising claims manually annotated with labels reflecting their check-worthiness. then we manually analyse the misclassifications of each model and provide an error analysis. we believe that the contributions of this work have important societal implications. the case we study here sheds lights on the future directions of claim detection tools for fact-checking based on pre-trained language models for low to medium resourced languages. this would contribute to the fight against dis/misinformation by scaling and speeding up the fact-checking process. the rest of the paper is organized as follows. in section 2 we give an overview of previous work on automated claim detection. section 3 describes the dataset and our experimental setup. we present and discuss our results and provide an error analysis in section 4. finally, we summarize our main findings, and discuss possible future works in section 5. ## background automated claim detection for fact-checking does not have a long history, but it has turned to be one of the attractive fields of research in nlp @xcite @xcite @xcite . one of the first studies on claim detection for afc is initiated as part of the claimbuster project @xcite . their initial claim detection system was based on a set of features (sentiment, word count, part of speech (pos) tags and named entities (ne)) followed by a feature selection and a traditional classifier namely naive bayes, svm, and random forest @xcite . claim detection has also been addressed in languages other than english. claimrank is a claim detection system that supports both arabic and english @xcite . a comprehensive set of features such as tf-idf, assertiveness, subjectivity, word embeddings are added to the claimbuster features and are fed to a two-layered neural network classifier @xcite . in recent years, employment of pre-trained language models (lms) in automated claim detection has been considered by numerous researchers @xcite @xcite . several instances of these works are presented in the check-worthiness detection sub-tasks in clef checkthat! editions (introduced in 2018 and ongoing) @xcite . checkthat! provides data sets in different languages (english, turkish, arabic, bulgarian, and spanish) for the claim detection task on twitter and political debates. the teams participating in this task have proposed classifier models mostly based on lms. for instance, the top-ranked teams in checkthat! 2020 used bert @xcite and roberta @xcite with enhanced generalization capability @xcite to detect check-worthy tweets. for the task of detecting claims in political debates, the baseline bilstm @xcite model with glove embedding outperforms the lm-based systems @xcite . claimhunter @xcite is another bert-based claim detection system that leverages xlm-roberta foot_1 @xcite , a multilingual version of roberta. it has been proved that the proposed model is superior to the classical baseline models nnlm+lr (neural-net language models embedding+logistic regression) and tf-idf+svm. to deal with the problem of small training data for lms, data augmentation is employed. claim detection from twitter has been approached by generating synthetic check-worthy claims with lexical substitutions using bert-based embeddings @xcite . this approach improves the performance of bert @xcite and roberta @xcite classification models @xcite . it has also been shown that the bertweet (nguyen et al., 2020) model, fine-tuned on claims normalized and augmented by substitutions using wordnet, surpasses a reference n-gram model @xcite . ## conclusion in this work, we conduct a case study using norwegian pre-trained lms for the task of automated claim detection. four existing norwegian models in addition to an svm baseline system are examined and compared using a claim detection data set that resembles a real world problem. the results show that language models outperform the baseline system. different models can be selected for different purposes. if the overall performance is to be prioritized, the norbert2 model is the best performing. if the recall is the focus, then the biggest nb-bert large model is to be selected. most of our observations can also be due to the differences between the lms. the behaviour of our models can be due to model architecture, training procedures, and the datasets they were originally trained on. we also show how the length and the source of the claim plays a role in prediction patterns. we believe that there is more that can be uncovered from the behaviour of these models, and we plan to explore this in future works.