D e T ox at G erm E val 2021: Toxic Comment Classification (2021)

Abstract: In this work, we present our approaches on the toxic comment classification task (subtask 1) of the GermEval 2021 Shared Task. For this binary task, we propose three models: a German BERT transformer model; a multilayer perceptron, which was first trained in parallel on textual input and 14 additional linguistic features and then concatenated in an additional layer; and a multilayer perceptron with both feature types as input. We enhanced our pre-trained transformer model by re-training it with over 1 million tweets and fine-tuned it on two additional German datasets of similar tasks. The embeddings of the final fine-tuned German BERT were taken as the textual input features for our neural networks. Our best models on the validation data were both neural networks, however our enhanced German BERT gained with a F1-score = 0.5895 a higher prediction on the test data.

## introduction in recent years social media platforms became a popular medium to discuss all kinds of topics with people around the world. also shops, companies, tv-shows and many more use social media to present their content to followers and discuss it with them. as it is possible to interact almost anonymously on the internet, such social media pages are often confronted with the problem of hate speech and toxic comments targeting single persons or whole groups @xcite . although hate speech detection has been a top research topic for several years, there exists no satisfactory solution yet @xcite . the germeval shared task 2021 @xcite addresses this topic -especially the side of social media moderators that are responsible to filter such comments -in this years challenge with the following three tasks, where we participate in subtask 1: therefore, we decided to experiment with standard supervised machine learning models and neural networks, different word embeddings, and pretrained transformer models. we then chose our best performing transformer model, enhanced it with re-training on extracted tweets in german, and finetuned it with additional datasets. the extracted word embeddings by our transformer model were used as an textual input for our neural network architectures besides additional features. our presented work is structured as follows: section 2 gives an overview of related work. in section 3 we describe the germeval 2021 data and the additional data we used for our final models. in section 4 the feature extraction, the baseline and the final models are described. in section 5, we show our final results and discuss our models. toxic speech can be defined as a combination of hate speech and offensive language (d'sa et al., 2020) or a type of aggressive writing style @xcite . many recent research uses deep neural networks for such detection tasks in social media content @xcite . there has also been some research with transformer models, especially for english social media content. maslej-krešňáková et al. (2020) compared multiple transformers and neural networks for the classification of toxic content with different types of preprocessing steps, focussing on word embeddings. however, some related work to our modelling approach has been done by researchers in similar content detection tasks on social media. sohn and lee (2019) used, in their study on hate speech detection with transformer models, a similar approach to our proposed models, after they finetuned a multi-channel bert model: they applied a dropout on the [cls] token of bert and added a feed forward layer before the softmax output and calculated the weighted sum of three transformers instead of only one. the [cls] token is the final hidden vector of bert used for classification, however it can also be extracted for the models embeddings @xcite . this was also done in @xcite for the task of automatic sexism classification, where the authors added features with a feed forward layer on top, however this did not improve their results. they also -in comparison to our concatenation strategy for our multilayer perceptron -created a bi-lstm (bidirectional long-short-term-memory), where they concatenated the additional extracted features (in this case user and network information) after going through several layers of the neural network with only using textual input. their work showed that using pre-trained embeddings for neural networks pushes the final classification by 3% @xcite . the study of @xcite found that using pre-trained models as an input for neural networks leads to better results than using complex deep neural networks or transformers as a stand-alone architecture. comparingly, another approach by d'sa et al. (2020) on hate speech detection analyzed fasttext @xcite and bert embeddings and used them as the input for deep neural networks without any additional feature ex- those studies show that combining transformers that are fine-tuned for a specific nlp task with neural networks is a promising approach to create better models for predicting toxic comments. since transformers are usually only used for training on the textual input, the feed forward layers can be concatenated with more extracted features. ## data in this section we describe the germeval 2021 shared task dataset as well as the supplementary datasets that we used for fine-tuning our model. ## methodology in this section the feature extraction methods as well as the baseline we used for comparison, the conducted preprocessing steps, and final models are described. our baseline models include different combinations. ## results and discussion all of our models were evaluated with precision, recall, and a macro-averaged f1-score as shown in therefore, we explored whether we set the threshold too high for our predictions on the test data. even though we experimented with setting the threshold to different values, we found that the predictions did not improve significantly (only ≈ 0.01), which shows that the neural networks probably overfitted on one class. we suspect this is also the reason for the very high validation recall in comparison to the precision. we plotted the confusion matrix for each model, shown in ## conclusion in this work we presented our submitted models for the germeval shared task 2021 on toxic comment classification. we decided to combine standard supervised methods with transformers and textual features, and to enhance the models with additional training data. our best model was a german bert that was re-trained on over 1.5 million additional german tweets from the first half year of 2019 and finetuned with two augmented datasets from similar tasks, such as hate speech and offensive language detection, as well as the germeval 2021 training data. even though our two multilayer perceptronswhich were trained on the extracted word embeddings by our transformer -showed better evaluation results during validation, our bert model still had a more robust prediction on the test set. for future work, we will further explore the combination of sequential neural networks and word embeddings by transformers and test several extraction and concatenation strategies.