UR @ NLP _ A _ T eam @ G erm E val 2021: Ensemble-based Classification of Toxic, Engaging and Fact-Claiming Comments (2021)

Abstract: In this paper, we report on our approach to addressing the GermEval 2021 Shared Task on the Identification of Toxic, Engaging, and Fact-Claiming Comments for the German language. We submitted three runs for each subtask based on ensembles of three models each using contextual embeddings from pre-trained language models using SVM and neural-network-based classifiers. We include language-specific as well as language-agnostic language models – both with and without fine-tuning. We observe that for the runs we submitted that the SVM models overfitted the training data and this affected the aggregation method (simple majority voting) of the ensembles. The model records a lower performance on the test set than on the training set. Exploring the issue of overfitting we uncovered that due to a bug in the pipeline the runs we submitted had not been trained on the full set but only on a small training set. Therefore in this paper we also include the results we get when trained on the full training set which demonstrate the power of ensembles.

## introduction the need to check and moderate conversations and text on social media keeps increasing proportionally to the use of social media over the years @xcite @xcite . research into the identification of hate speech or toxic comment and fake news have recently become more popular in languages other than english because the abuse of free speech online and spread of information whether false or true extends farther than we can imagine @xcite . germeval 2021 @xcite contains three subtasks not only aimed at identifying toxic comments in german text on social media platforms like in previous years @xcite but also the classification of engaging and fact-claiming comments. in a way to help the situation of diffusing toxic content and promote positive content moderators on popular social media platforms also seek to promote texts that engage other users in a healthy conversation @xcite . the connection between hate speech and fake news is immense as the latter can rather stir up the masses into targeted hate towards a group of people or in some instances deadly violence @xcite . therefore identifying social media content that makes a-need-to-check claim is as important as identifying hate content online. our participation in germeval 2021 was in all three subtasks and involved the use of the same model architectures on all three to learn, compare and analyse how models behave on subtasks. we applied transformer-based embeddings (bert), rnn-based embeddings (bilstm) with a classifier either utilising a densely connected output layer of a simple neural network or a support vector machine in an ensemble constructed with majority voting of three models on all three subtasks. the next sections discuss in detail the dataset used for our experiment and the model architectures applied. we also discuss and compare the performances of the models on the subtasks. all code used in this experiment can be accessed via github. foot_0 2 dataset and task the dataset provided for this competition includes a trial set of 113 user comments, a training set of 3,244 user comments and a test set of 944 user comments of german text in csv format. the training set provided consists of over 3,000 facebook anonymized user comments that were annotated by four trained annotators @xcite . the dataset was extracted from the home feed of the facebook page of a political talk show of a german television broadcaster as well as the comment section discussions of posts from the same page from july 2019 till february 2021. it was shared in fully anonymized form and no user information or comment ids were revealed. links referring to users were replaced by @user, links referring to the show were replaced by @medium, and links referring to the moderator of the show were replaced by @moderator. the csv file contained all comments and labels for all 3 subtasks. that is to say, a user comment can be either toxic, engaging, fact-claiming or any of 2 of the labels or all 3 or neither of the labels (see figure 1 ). ger-meval 2021 consists of 3 subtasks @xcite . the first subtask is the identification of toxicity or hate speech from german text. the second and the third are the identification of engaging text and fact-claiming text, respectively. participants were to choose any or all of the tasks they would participate in. we participated in all 3 tasks using a system of 3 different ensembles for each task (see figure 2 for a quick overview). submissions of the runs were submitted to codalab. ## models architecture over the past few years, long short-term memory (lstm) @xcite and pre-trained transformer-based models @xcite ) have proven to be effective in various nlp tasks through their ability to generate word or sentence embeddings @xcite . one of such models that have widely been used in many nlp tasks is the bidirectional encoder representations from transformers (bert) @xcite . it is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context to learn and produce embeddings either on sentence or word level in a transformer based architecture. the flair embedding architecture is also an example of a model that uses a variant of bidirectional recurrent neural networks (bilstms) with a conditional random field (crf) layer to generate contextual embeddings from both directions @xcite . ## results the results of our officially submitted runs are displayed in we subsequently re-run the three approachesthis time trained on the full training set -as illustrated in the results demonstrate that, as expected, an increase in the training data has a measurable positive effect on the overall performance across all metrics. the results recorded after training shows that the svm models had very high metrics on the trial set whereas the ann models had relatively low metrics peaking at 62% for f1 score, precision and recall. an ensemble approach rather seemed bal-anced. the ensemble models were slightly biased towards the svm models because in a total of three models for each ensemble, two models were svm models for both ensemble 1 and 2. ensemble 3 was a model of 3 svm models. it is fair to say that the svm models were overfitted on the trial set. the results from the test set were lower than the results for the training data (see most interesting are of course the findings we derive from running our three approaches on the full training data. we observe robust performance of our ensemble-based approaches. we also observe that fine-tuning one of the models in our ensembles appears to push up performance quite substantially. ## conclusion ensemble approaches have repeatedly been shown to offer great benefits but they nevertheless rely on good underlying individual models. in our runs we combined contextual embeddings using state-ofthe-art models such as bilstm-crf, bert-based models and svm and simple neural networks as classifiers in an ensemble approach to perform binary text classification in german. we observe robust performance across different tasks, we also note a positive impact of including fine-tuned models in our ensembles.