UPA pplied CL at G erm E val 2021: Identifying Fact-Claiming and Engaging F acebook Comments Using Transformers (2021)

Abstract: In this paper we present UPAppliedCL’s contribution to the GermEval 2021 Shared Task. In particular, we participated in Subtasks 2 (Engaging Comment Classification) and 3 (Fact-Claiming Comment Classification). While acceptable results can be obtained by using unigrams or linguistic features in combination with traditional machine learning models, we show that for both tasks transformer models trained on fine-tuned BERT embeddings yield best results.

## introduction in the last decade social media platforms, like facebook foot_0 , have gained a notable momentum, which is reflected by the increasing number of users of social media. 2 while facilitating communication across the globe, from the perspective of nlp however, systems need to be specifically adapted to social media for the following reasons. first, social media data is unedited and contains certain conventions which can pose challenges for systems trained on more well-formed texts @xcite . second, social media platforms are used for different kinds of communication ranging from everyday conversations to sophisticated evidence-based argumentation on political issues. while the latter have the potential to contribute to public political discourse in general, social media has been found to contain not only respectful and engaging discussions but also hateful speech, which threatens the respectful exchange and possibly also the mental well-being of its participants. the germeval 2021 shared task @xcite aims to stimulate research on this issue, while also going beyond the single task of toxic comment classification. in this paper we present upappliedcl's contribution foot_2 to the germeval 2021 shared task which consists of three subtasks revolving around the mentioned characteristics of social media discussions: 1. toxic comment classification; 2. engaging comment classification; 3. fact-claiming comment classification. here we especially focus on subtask 3 (fact-claiming comments), which is also relevant for tasks in the field of argument mining (am) @xcite . in addition, we also participate in subtask 2 (engaging comments), which we consider as a first albeit facultative step in an am system in order to identify potential argumentative comments. as we consider subtask 1 (toxic comments) as a task which is more independent of am, we will not attend to it in this work. the paper is structured as follows: in section 2 we give a short overview of relevant previous work. we present the dataset provided by the organizers in section 3. in section 4 we describe our approach including the developed baselines, and in section 5 we continue with the obtained results, which are discussed in section 6. we conclude the paper in section 7. ## related work given that we do not participate in subtask 1 (toxic comments) we will not go further into details here. for surveys on tackling this issue using nlp techniques we refer the reader to @xcite and @xcite . subtask 2 (engaging comments) may be seen as a complement task to toxic comment classification as it focuses more on the identification of respectful conversation. approaches include work by @xcite who propose a system that relies on upvotes and replies in order to identify news comments that potentially attract user engagement. a neural network model obtained classification accuracies ranging from 0.68 to 0.72. subtask 3 (fact-claiming comments) can be approached from the perspective of am, i.e., identifying fact-claiming content can be an important first step for further proving its actual correctness. related work was published by @xcite who investigated the classification of factual and opinionated tweets, which is defined as a pre-task for later checks of correctness, e.g., via source identification. a logistic regression model trained on a set of lexical, twitter-specific, syntactic/semantic and sentiment features yielded an f1 score of 0.80. note, however, that no information is given whether micro or macro f1 scores are reported. factual information can also be used as evidence for claims. in that sense, fact-claiming comment classification can be interpreted as a pre-task for evidence detection, which has previously been investigated for different text sources including social media. for instance, in our previous work, we investigated different am tasks including evidence detection on an expert and crowd annotated german tweet dataset. to this end we used classification and sequence labeling techniques. for evidence detection on the expert annotated dataset we obtained macro f1 scores of 0.60-0.75 for classification (xgboost) and 0.61-0.72 for sequence labeling (crf) @xcite . ## data the provided training set consists of 3244 german comments, which were collected from the facebook page of a german political talk show. the comments were posted from february to july 2019 on two shows. all comments were anonymized. this includes replacement of user links with @user, show links with @medium and moderator links with @moderator. the comments were annotated by four trained expert annotators. for measuring inter annotator agreement (iaa) the krippendorff's α metric was used. in total three binary annotation layers were created, each for one of the three subtasks of germeval 2021. toxic comments: toxic comments include different types of uncivil behavior like insults, sarcastic language, discrimination, and threats of violence. it also comprises attacks on democratic principles (iaa: 0.73 < α < 0.90). engaging comments: engaging comments comprise language centering around rationality, mutual respect, empathy for others and their standpoints, and mediation (iaa: 0.71 < α < 1.0). fact-claiming comments: fact-claiming comments focus on the assertion of facts, or evidence provided by external sources (iaa: 0.73 < α < 0.84). for the development of our system we conducted a stratified split on the provided training set in order to obtain training, development and test sets. both development and test set consisted of about 12.5% of the former training set. we used the development set to experiment with different feature sets and hyperparameters, while the test set was only used to calculate the preliminary test results presented in this paper. for final system evaluation, 944 additional unlabeled comments were provided. these were drawn from discussions on a different show to avoid a topical bias. ## system description in this paper we follow a machine learning (ml) approach based both on traditional ml methods and more recent deep learning (dl) techniques. we define three baselines against which we compare our submitted systems. all systems are evaluated using macro f1, precision and recall scores. ## results our results are based on two different datasets: 1. the test set that we obtained from our own splitting submission features classifier i bert emb (ft) transformer ii bert emb transformer iii bert emb xgboost ## discussion in this section we discuss some of the results obtained by the submitted models. as shown in section 5 transformers trained on fine-tuned bert embeddings yield best f1 scores, which indicates that fine-tuning does not lead to overfitting. this is the case for testing with the in-domain testing set, evaluating with the final evaluation set and for both subtasks. further, this is in line with our ranking hypothesis. interestingly, however, an xgboost model performs better on the test set of subtask 2 than a transformer if both are trained on non-fine-tuned bert embeddings, which contradicts our ranking hypothesis. in contrast, a transformer is more successful than an xgboost model on subtask 3. model differences on the evaluation set, however, are less substantial. evaluation f1 scores on subtask 2 are equal. it is difficult to argue why these patterns arise. however, from these results we can carefully conclude that dl models like transformers do not necessarily outperform traditional ml models. from it. this may have interesting implications with respect to the application's focus. the results suggest that a model needing a high recall can benefit from embedding fine-tuning, while ml practitioners requiring a higher precision may refrain from fine-tuning. this finding, of course, requires more investigation before making generalisations, especially as it is less pronounced in the evaluation results. scores yielded for subtask 3 tend to be higher than for subtask 2. we argue that this might be related to the class distribution, which is more balanced in subtask 3. scores obtained by evaluation are lower than by testing. this, however, is expected due to the different topics covered in training and evaluation data. recall that the test data is topically closer related to the training set than the evaluation set. given that we still achieved good results, especially for subtask 3, we argue that our models are capable of solving both tasks to a promising degree. ## conclusion in this paper we presented approaches to factclaiming and engaging comment classification. we applied different combinations of features (unigrams, linguistic features, bert embeddings) and classification algorithms including more traditional ml techniques like svm, rf or xgboost and more recent dl techniques like transformer models. our experiments show that best results can be achieved by using fine-tuned bert embeddings in combination with a transformer. we also found that fine-tuning leads to a higher recall while precision benefits from refraining from fine-tuning. as this pattern is less obvious in the evaluation set we do