ur-iw-hnt at G erm E val 2021: An Ensembling Strategy with Multiple BERT Models (2021)

Abstract: This paper describes our approach (ur-iw-hnt) for the Shared Task of GermEval2021 to identify toxic, engaging, and fact-claiming comments. We submitted three runs using an ensembling strategy by majority (hard) voting with multiple different BERT models of three different types: German-based, Twitter-based, and multilingual models. All ensemble models outperform single models, while BERTweet is the winner of all individual models in every subtask. Twitter-based models perform better than GermanBERT models, and multilingual models perform worse but by a small margin.

## introduction moderation of popular social media networks is a difficult task. facebook alone has almost 2.8 billion active users on april 2021 @xcite . moderating discussions between users simultaneously all day is an impossible task, so moderators need help with this work. also, fully automated solutions for content moderation are not possible, and human input is still required @xcite ). an ai-based helper solution for harmful content detection is needed to make social networking less toxic and more pleasant instead. the shared task of germeval2021 focuses on highly relevant topics for moderators and community managers to moderate online discussion platforms @xcite . the challenge is not to specialize in one broad nlp task like harmful content detection but to detect other essential categories like which comments are engaging or factclaiming. we participated in all three subtasks (toxic, engaging and fact-claiming comment classification) to test our ensemble model to see whether multiple bert-based models provide robust performance for different tasks without further customization. moderators would benefit from a working system without having to change models or settings all the time. this report discusses in detail the three runs we submitted in the germeval2021 shared task @xcite . we start with a brief reflection on related work, only focussing on aspects that are closely aligned with the subtasks. we then explain the dataset and the shared tasks in more detail. next, we present our experiments, some discussions of the results, and we finally draw some conclusions. to encourage reproducibility of experimental work, we make all code available via github foot_0 . ## related work detecting harmful content in social media platforms is not only a monolingual but a multilingual issue. a multilingual toxic text detection classifier uses a fusion strategy employing mbert and xlm-roberta on imbalanced sample distributions @xcite . deep learning ensembles also show their effectiveness in hate speech detection @xcite . a taxonomy of engaging comments contains different possible classifications @xcite . with the increasing spread of misinformation, more collaborations with it companies specialized in factchecking and more intelligence and monitoring tools are available to help to identify harmful content @xcite ). an attempt to fully automate fact-checking is the tool called claimbuster @xcite . another tool named crowdtangle monitors social media platforms and alerts the user if specific keywords are triggered so manual factclaim checking can be done @xcite . in addition, an annotation schema for claim detection is also available @xcite . the dataset for the shared task of germeval2021 consists of 3,244 annotated user discussion comments from a facebook page of the german news broadcast in the timeframe of february to july 2019, labeled by four annotators in three different categories for binary classification: toxic comments, engaging comments and fact-claiming comments @xcite . since the labels are imbalanced, we first applied a stratified split onto the dataset so that 80% is for training. we then again apply a stratified split on what is left into two halves, the first part is the development set, and the second part is the holdout set for evaluation which we call the evaluation set here. after training, the ensemble strategy predicts the test dataset, consisting of 944 comments. ## discussion our experiments demonstrate that bertweet was showing better performance than every other model in every subtask, which is a surprise. we expected the monolingual germanbert models to perform best because of the cultural context in the integrated german language. multilingual bert models perform worst but by a close margin. because of an overfitting issue, the tenth bert classifier xlm-r performed faultily, only recognizing negative labels and thus the low macro-averaged f1 scores. the margin of each ensemble performance in subtasks 1 and 3 is around 1%, and for subtask 2 only around 2%. we conclude that the ensembling strategy shows robustness and consistency for the choice of good classifiers in a big enough amount for each task, and it could be a legitimate approach for the overfitting problem. because of time constraints, no cross-validation was conducted, and since the holdout set was chosen not to be released for training, there is still improvement in the training quality of the bert models so that more experiments are needed. each part of a system like the gpu influences the training accuracy, so an identical replication is difficult to achieve, leading to different results. that is why reproducibility is not guaranteed, even if a manual seed is set foot_6 . also, the amount and the imbalance of the dataset can lead to overfitting and lower scoring. ## conclusion and future work we presented an ensemble strategy using ten bert classifiers, including the use of machine translation, demonstrating robustness across tasks. while ensembles perform best overall, twitter-based models (using standard bert hyperparameter values) with translation to english perform best in a single model setting. this observation might change if cross-validation, early stopping, hyperparameter tuning, and other optimization techniques for each model are available for future work.