H unter S peech L ab at G erm E val 2021: Does Your Comment Claim A Fact? Contextualized Embeddings for G erman Fact-Claiming Comment Classification (2021)

Abstract: In this paper we investigate the efficacy of using contextual embeddings from multilingual BERT and German BERT in identifying fact-claiming comments in German on social media. Additionally, we examine the impact of formulating the classification problem as a multi-task learning problem, where the model identifies toxicity and engagement of the comment in addition to identifying whether it is fact-claiming. We provide a thorough comparison of the two BERT based models compared with a logistic regression baseline and show that German BERT features trained using a multi-task objective achieves the best F1 score on the test set. This work was done as part of a submission to GermEval 2021 shared task on the identification of fact-claiming comments.

## introduction deception and misinformation have become increasingly common on social media @xcite . with the endless cycle of production and consumption of information, social media users may be influenced to make choices which are unsafe for them and for society. as a result, misinformation media has attracted the interest of the nlp community to develop methods to combat it. since information on social media is often not filtered based on veracity, it is necessary to apply fact-checking in order to verify the claims made in social media posts and comments @xcite . in this paper, we address the problem of automatic fact-claiming comment detection in german, which is a crucial aspect of the fact checking process. fact check-worthiness detection has been studied in prior work by using naive bayes and svm 1 code is available at https://github.com/ subhadarship/germeval2021. in this work we use contextualized embeddings from multilingual bert and german bert for detecting fact-claiming comments. we also train our model on two auxiliary tasks of toxicity and engagement prediction in a multi-task learning setup. upon comparing the methods, we find that the multitask model based on german bert achieves the best macro-average f-1 score on the test set. we outline the remaining part of the paper by first describing the dataset used for our experiments in section 2. then we discuss the approaches we use in section 3. sections 4 and 5 contain details of the experiment setup and the results obtained. section 6 concludes the paper. ## data we use the data provided for the germeval 2021 shared task @xcite . the sizes of the training and test splits along with the number of samples in each class are shown in ## methodology the task is to classify a given comment as factclaiming or not. it is a binary classification task. ## experiments in this section we outline the baseline experiments using logistic regression model and experiments using bert-based models. ## results the predictions were evaluated using the macroaverage f1 score. for the dev set results, the macroaverage f-1 score is computed using the implementation provided in scikit-learn whereas for test set results the macro-average f-1 score is computed using the evaluation script provided by the organizers of the shared task. 3 the difference between the two is that while the former computes the arithmetic mean of the f-1 scores for each class, the latter computes the arithmetic mean of the precisions and the arithmetic mean of the recalls for each class which are then used to compute f-1 3 link here. score using f 1 = 2×p ×r p +r . foot_1 ## conclusion in this paper we presented a baseline model based on logistic regression and two bert-based models for identifying whether german comments in social media are fact-claiming or not. we thoroughly compared the two models: multilingual bert which is pre-trained on 104 languages, and german bert which is pre-trained only on german data. we also formulated the learning problem as a multitask problem by addition of two auxiliary tasks of toxicity and engagement classification. the multitask german bert achieved the best results on the test set. this work contributes models and insights for detecting fact-claiming comments on social media, which is an important step towards hopefully combating misinformation that is pervasive on social media.