FH - SWF SG at G erm E val 2021: Using Transformer-Based Language Models to Identify Toxic, Engaging, & Fact-Claiming Comments (2021)

Abstract: In this paper we describe the methods we used for our submissions to the GermEval 2021 shared task on the identification of toxic, engaging, and fact-claiming comments. For all three subtasks we fine-tuned freely available transformer-based models from the Huggingface model hub. We evaluated the performance of various pre-trained models after fine-tuning on 80% of the training data with different hyperparameters and submitted predictions of the two best performing resulting models. We found that this approach worked best for subtask 3, for which we achieved an F1-score of 0.736.

## introduction compared to the detection of offensive language in germeval 2018 @xcite and @xcite , this year's task adds two important additional categories found in social media comments, namely engaging and factclaiming comments @xcite . with federal elections being held in 2021, identifying fact-claiming statements (subtask 3) in german social media posts has gained additional relevance as "fake news" might have had an influence on other important elections, e. g. the 2016 us presidential elections @xcite . a system identifying fact-claiming comments could help to identify potential attempts to spread false factual statements. the identification of engaging comments (subtask 2) is potentially interesting for the ranking algorithms used by social network providers. increasing the visibility of these comments might help improving the attractiveness of a social network by encouraging the users to employ a more respectful and rational style of discussion. with the classification of toxic comments (subtask 1), the germeval shared tasks on the iden-tification of offensive language mentioned above are continued. this category is also useful for the ranking algorithms of social media providers and could be used to decrease the visibility of such comments. however, we have made the experience that this year's toxic category is harder to identify than the former offensive categories -at least by our approach. the best performing systems in germeval 2019 were based on bert @xcite . leveraging the transformer architecture @xcite with its attention mechanism, bert is able to model relations between words and to create semantic embeddings of sentences @xcite . in the last two years, various modifications of bert like roberta @xcite or elec-tra @xcite have been proposed and shown to achieve state-of-the-art results on various nlp tasks. other transformer-based models, especially gpt-2 @xcite and its successor gpt-3, even made it into the press @xcite due to their ability to create high-quality artificial text or to create source code for various programming languages @xcite . probably the most important feature of these models is that they allow transfer learning: after an unsupervised pre-training, the resulting models can be fine-tuned for various nlp tasks like token classification (e. g. ner) and sequence classification. pre-training a language model for german imposes two challenges: it requires a large corpus of text and is computationally expensive. according to @xcite , gpt-3 was trained on a corpus of 400 billion byte-pair-encoded tokens or roughly 570 gb of text. compared to this, the "huge german corpus" foot_0 with 204 million tokens is rather small. bert-large was trained on 64 tpu chips for four days at an estimated cost of $7,000 @xcite , the training of gpt-3 took 3.640 petaflop-days @xcite . due to the high computational effort and costs to train a model from scratch, we decided to evaluate freely available pre-trained models for our system. for english, pre-trained models of high quality are freely available for most of the model architectures mentioned above (with the notable exception of gpt-3). unfortunately, the groups which developed and trained these models and the companies behind them do not deem german important enough to provide pre-trained models for german. although there is currently no active academic community in germany training and publishing these language models, there is a growing number of companies and individuals publishing such pre-trained models. for example, deepset.ai has published a german electra model achieving an f1-score (macro average) of 80.70% on ger-meval 2018 coarse and an f1-score (micro average) of 88.95% on germeval 2014 @xcite . philipp reissel and philip may have published both a german electra model @xcite and a "german colossal, cleaned common crawl corpus" (gc4) @xcite with about 540 gb of german text from the web it would be helpful for the development of language models for german if an extensive and high-quality corpus of german language text would be available through infrastructure projects like clarin-d @xcite . ## setup our experiments were performed using jupiter notebooks @xcite . this had the advantage that we could use local computing resources and cloud platforms like google colaboratory (bisong, 2019) without modifications to the code. the code used to generate our submissions is available on github foot_2 . we used the web application weights & biases @xcite to record and compare the results of experiments with different language models and hyperparameters (learning rate, number of training epochs), which was of great help especially when using cloud-based computing resources without a persistent storage medium. ## model library a large repository of pre-trained transformer based language models along with an open-source library of implementations of them is operated by huggingface @xcite . as of july 2021, the model hub contains about 2,900 pre-trained models for english and more than 200 pre-trained models for german provided by a fast-growing number of contributors, including the groups mentioned above. due to the large number of available pretrained models for german, we decided to use the huggingface transfer library for our submission and to choose among the models available on the model hub. the transformer library makes it very easy to use and to fine-tune the models provided on the hub. besides the model implementations, it also contains recent optimization algorithms like adamw @xcite and adafactor @xcite , provides integration with the experiment-tracking software weights & biases @xcite , code for loading and handling training data, and commonly used metrics. ## data preprocessing the transformer-based language models we used for our experiments use either sentencepiece @xcite or byte pair encoding @xcite for tokenization and can handle rare words and emojis. so we did actually not preprocess the texts in any way. one of the models we used in our experiments, german-nlp-group/electrabase-german-uncased, is an uncased model that converts all characters to lower case during tokenization. unlike other 'uncased' models published on the model hub, this model does not remove accents. ## model selection with more than 200 pre-trained models for german available on the model hub, we needed to do some preselection for our experiments. philip may, one of the authors of german-nlpgroup/electra-base-german-uncased, has evaluated several models on the germeval 2018 dataset (see figure 1 ). we chose the best three models of this evaluation as our candidates. due to the success of gpt-2 on various nlp tasks @xcite , we also included benjamin/gerpt2-large, a german gpt-2 model recently published by minixhofer (2020), an ai student from johannes kepler universität linz. the following list contains some information on these models. since we are not sure how to calculate the number of model parameters from the specification in the model configuration file, we specify the size of the binary file containing the model parameters as a measure of model complexity. the transformer library provides the class gpt2forsequenceclassification for this purpose. with a size of 3.2 gb it is the largest model we used. ## computing resources most calculations were done on a local server using a tesla v100s gpu card. we used fp16 precision for the training runs on the v100s for better performance as some tests with double precision did not show better results. in addition, we used cloudbased computing resources provided by graphcore and google colaboratory. ## results using the four models (see section 5) we performed several training runs with a train-test split of 80%. we did not have the time and computing resources to do a systematic hyperparameter optimization but rather tried different learning rates and number of training epochs. figures 2 3 4 show the resulting f1-scores of several runs ans models for the three subtasks. unfortunately, the fluctuations of the f1scores measured on the 20% test split during the training were about as large as the differences between the different models. at this point, we would have needed more time and resources to perform a larger number of training runs and a statistical analysis similar to the one shown in figure 1. in some runs, declining f1-scores at the end of the training runs indicated overfitting -additional training data would probably have improved the results. overall, we achieved the best results by fine-tuning deepset/gelectra-large and bjamin/gerpt2-large. for the final system submissions, we fine-tuned these two models using the complete training dataset for all three subtasks. ## using additional training data assuming that offensive language is also considered toxic, we tried to add data from germeval 2018 and 2019 to our training dataset for subtask 1. however, compared to experiments without this additional training data, accuracy and f1-score on our validation dataset (i. e. 20% of this year's training data) were worse for these experiments. at least for an ai, toxic comments on facebook seem to be quite different from offensive language used on twitter. ## error analysis before the gold labels were released, we compared our model predictions with our personal predictions for the first test comments. when we looked at the gold labels, we were surprised by some of the labels, especially with respect to examples having more than one label. for example, our system flagged a fact claim in comment 3246 @user , ich glaube,sie verkrnnen gründlich die situation. deutschland mischt sich nicht ein, weil die letzte einmischung in der ukraine noch nicht bereinigt ist. es geht nicht ums militär which we considered correct. we did not expect that this comment is also considered engaging. als jemand, der im real existierenden sozialismus aufgewachsen ist, kann ich über george weineberg nur sagen, dass er ein voll...t ist. finde es schon gut, dass der eingeladen wurde. hat gezeigt, dass er viel meinung hat, aber offensichtlich wenig ahnung. er hat sich eben so gut wie er kann, für alle sichtbar, zum trottel gemacht. on the other hand, comment 3269 sry aber preetz hat nicht viel beizutragen. er muss der politik in den hintern kriechen damit sein verein zuschauer ins stadion bekommt. er ist abhängig von der politik. these three examples demonstrate that this year's task is really hard -even for humans. it would be interesting to measure the score of human annotators getting just the category names and the training examples. when we first looked at the development data, our impression was that fact-claiming statements would be the hardest category to recognize for an nlp system due to the wide range of different facts in the statements. the rather low range of annotator agreement of 0.73 < α < 0.84 for subtask 3 also suggests that this should be the "hard" category. we were quite surprised that our system actually achieved the best f1-score (0.736 in the case of benjamin/gerpt2-large) for this category. regarding the toxic category, the f1-score of 0.707 on subtask 1 is surprisingly low considering the f1-score of deepset/gelectra-large of about 0.80 reported by @xcite . this year's 'toxic' category seems to be quite different from the offensive language category of the germeval tasks in 2018 and 2019 and -at least for an ai -more difficult to recognize. the approach we used to create our submissions is a rather simple one that did not require preprocessing of the training data or much programming. free libraries containing implementations of a wide range of language models and the availability of an increasing number of pre-trained model instances make it quite easy to apply state-of-the-art language models for nlp tasks like text classification. it still, however, requires some coding to train and select models and to create predictions for the test dataset. integrated tools like the recently announced au-tonlp 3 will probably enable non-experts (and noncoders) to train such models in the next few years.