FHAC at G erm E val 2021: Identifying G erman toxic, engaging, and fact-claiming comments with ensemble learning (2021)

Abstract: The availability of language representations learned by large pretrained neural network models (such as BERT and ELECTRA) has led to improvements in many downstream Natural Language Processing tasks in recent years. Pretrained models usually differ in pretraining objectives, architectures, and datasets they are trained on which can affect downstream performance. In this contribution, we fine-tuned German BERT and German ELECTRA models to identify toxic (subtask 1), engaging (subtask 2), and fact-claiming comments (subtask 3) in Facebook data provided by the GermEval 2021 competition. We created ensembles of these models and investigated whether and how classification performance depends on the number of ensemble members and their composition. On out-of-sample data, our best ensemble achieved a macro-F1 score of 0.73 (for all subtasks), and F1 scores of 0.72, 0.70, and 0.76 for subtasks 1, 2, and 3, respectively.

## introduction social media plays a role in the spreading of problematic content, ranging from conspiracy theories and concerted misinformation campaigns to offensive language in user comments @xcite . moderating comments remains a challenge due to the ever-increasing amount of user-generated content created daily. one promising approach to addressing this challenge are techniques from natural language processing (nlp) that support manual moderation processes by, for example, alerting human moderators to potentially problematic comments. among the many factors that have driven recent progress in nlp, we note in particular (i) methodological advances in language modeling and (ii) the availability of annotated data due to shared tasks. recent methodological advances can be traced back to the invention and availability of deep neural network models. a major contribution was the invention of the transformer architecture, which harnesses self-attention mechanisms to effectively model long-range correlations in series of tokens (e.g., sentences) @xcite . based on the transformer architecture, neural network models such as bert @xcite were proposed and trained in a self-supervised fashion on large unannotated text corpora. language representations learned by bert turned out to be effective in many downstream tasks, leading to new state-of-the-art nlp systems. while masked language modeling and next sentence prediction are used as objectives in self-supervised pretraining to learn representations in bert, other pretraining objectives such as replaced token detection (electra, @xcite ) have been demonstrated to yield language representations that can be better suited for various downstream tasks @xcite . furthermore, language representations have been learned in multilingual language models (such as mbert) and in language specific bert models @xcite . recent german specific language models include the bert-based models gbert @xcite and gottbert @xcite as well as the electra based model gelectra @xcite . the second factor driving progress in nlp has been recurring shared tasks that foster the exchange of ideas, the development and comparative assessment of methods, as well as the availability of annotated data @xcite . in addition to multilingual shared task campaigns (see, e.g., mandl et al. (2019 @xcite and distinguishing between profane, offensive, or abusive language found in twitter tweets @xcite . the germeval 2021 shared task on identifying toxic, engaging, and fact-claiming comments @xcite provided german comments from a facebook page of a political talk show of a german television broadcaster. in this contribution, we investigate the ability of ensembles of gbert and gelectra models to identify toxic, engaging, and fact-claiming comments. our work was inspired by previous studies on german bert models @xcite and ensemble approaches @xcite . we study the dependence of classification performance on the number of ensemble members and ensemble composition. finally, we describe the models that were evaluated in the germeval 2021 shared tasks and report performance scores achieved on out-of-sample data. the implementation details of our experiments are available online foot_1 . ## data and tasks the dataset of the shared task consisted of 3244 annotated facebook comments and was provided by the organizers of germeval 2021 @xcite . the comments were drawn from a facebook page of a political talk show of a german television broadcaster from february till july 2019 and were anonymized by replacing links to users by @user, links to the show by @medium, and links to the moderator of the show by @moder-ator. four trained annotators labeled the data by figure 2 : venn diagram showing the numbers of comments that were labeled as toxic, engaging, or fact-claiming. 33 % of all comments were not assigned any class, whereas 24 % were attributed to more than one class. the shared task consisted of three binary classification subtasks that aimed at predicting whether a given comment belonged to a category (class) or not @xcite . comments were considered toxic (subtask 1) when they could violate the rules of polite behavior or violated democratic discourse values. automated identification of such comments can be particularly valuable for managers of online communities. comments were considered engaging (subtask 2) when they were in line with deliberative principles such as rationality, reciprocity, and mutual respect. such comments might encourage user engagement and could be made more visible in online communities. finally, comments were considered fact-claiming (subtask 3) if they contained assertion of facts and/or provided evidence by citing external sources. identifying such comments may constitute a preprocessing step that could assist community managers to filter out misinformation and fake news. we did not observe any major class imbalance @xcite as all three classes occurred with similar frequencies in the dataset (35% toxic, 27% engaging, 34% fact-claiming). however, the venn diagram (see figure 2 ) demonstrated significant overlap between classes where 24% of all comments were attributed to more than one class. for instance, the large overlap between the engaging and the fact-claiming classes may point towards a correlation between these two classes. such label correlations can be exploited by multi-label classification approaches to improve classification performance @xcite . thus we pursued a two-fold strategy. (i) in our first approach, we trained a multi-label classifier to predict all the possible class attributions for a given comment. such models are called multi-label in the following. (ii) in our second approach, we trained separate binary classifiers that aimed at distinguishing between toxic and non-toxic, engaging and nonengaging, or fact-claiming and non-fact-claiming classes, respectively. this approach of transforming a multi-label classification task into multiple single-label classification tasks is also known as a binary relevance transformation @xcite . we call such models single-label in the following. note that in this case, training a size 30 ensemble to classify comments means training three separate size 30 ensembles, each making predictions for one of the three binary classification tasks. ## conclusion we trained ensembles of fine-tuned german language models, namely gelectra and gbert, to classify german toxic, engaging, and fact-claiming comments in the germeval 2021 shared task. we investigated whether classification performance (quantified by macro-f1 scores) depended on (i) ensemble size, (ii) ensemble composition, or (iii) whether models were trained as multi-label classifiers (and thus potentially exploiting label correlations) or as single-label classifiers. we observed that ensemble size had a significant effect on classification performance, with more ensemble members leading to better macro-f1 scores, consistent with previous observations by @xcite on a different dataset. neither ensemble composition nor model classification type (multi-or single-label) showed significant different classification performance for the studied parameters when the ensemble size was larger than 30. two ensembles achieved the largest macro-f1 score (0.73) on the test data, namely the multi-label and single-label ensembles consisting of gelec-tra and gbert models.