WLV - RIT at G erm E val 2021: Multitask Learning with Transformers to Detect Toxic, Engaging, and Fact-Claiming Comments (2021)

Abstract: This paper addresses the identification of toxic, engaging, and fact-claiming comments on social media. We used the dataset made available by the organizers of the GermEval2021 shared task containing over 3,000 manually annotated Facebook comments in German. Considering the relatedness of the three tasks, we approached the problem using large pre-trained transformer models and multitask learning. Our results indicate that multitask learning achieves performance superior to the more common single task learning approach in all three tasks. We submit our best systems to GermEval-2021 under the team name WLV-RIT.

## introduction the popularity and accessibility associated with social media have greatly promoted user-generated content. at the same time, social media sites have increasingly become more prone to offensive content @xcite @xcite . as such, identifying the toxic language in social media is a topic that has gained, and continues to gain traction. research surrounding the problem of offensive content has centered around the application of computational models that can identify various forms of negative content such as hate speech @xcite , abuse @xcite , aggression @xcite , and cyber-bullying @xcite @xcite . germeval-2021 @xcite focuses on identifying multiple types of comments in social media. this year's shared task is divided into three distinct classifications of comments: i) toxic, ii) engaging, and iii) fact-claiming. like previous germeval shared tasks @xcite , the detection of toxic content remains an integral part of germeval-2021. regarding engaging comments, there is an increasing desire from community managers as well as moderators to identify valuable user content @xcite . more particularly, rational comments that serve to encourage readers to engage in a discussion. in a similar light, identifying fact-claiming comments is equally important as platforms need to consistently review and verify user-generated content to uphold their responsibility as information distributors @xcite . we pose that multitask learning (mtl) is a suitable approach for this year's germeval as it enables what is learned from each task to aid in the learning of other tasks. the current state-of-the-art approach for offensive language identification is neural transformers modeled using single task learning (slt) @xcite . it is well-known that training large neural transformer models often result in long processing times. as germeval-2021 features three related tasks, from a performance standpoint, we pose that training a model jointly on three tasks is likely to be computationally more efficient than training three models in isolation. moreover, as germeval-2021 provides a single dataset for the three tasks, mtl can also be used to help improving performance across tasks. as such, we introduce multitask learning whereby one model can predict all three tasks as an alternative approach. in this paper, we present the methods and results of the wlv-rit submission to the germeval-2021 shared task. we explore transformer architectures in two different environments, single task learning and multitask learning, and describe them in detail in section 4. we perform several experiments using three transformer models that support german and evaluate their performance on the germeval-2021 dataset. the identification of offensive language in online discussions is an extensive topic that has become popular over the past several years. the majority of the research related to this topic is centered on english data due to the availability of annotated datasets @xcite . notwithstanding this, offensive language datasets are being annotated in other languages. researchers have examined offensive content across multiple social media platforms and have both annotated and utilized data from different languages such as greek @xcite , marathi @xcite , italian @xcite , portuguese @xcite , arabic @xcite , turkish (c ¸öltekin, 2020), and multiple languages of india @xcite . past approaches to tackling the problem of offensive content on social media have relied on using a variety of computational models ranging from traditional machine learning classifiers such as logistic regression and svms @xcite , to various deep learning models (de @xcite . semeval-2019 task 5 (hateval) @xcite presented the challenge of detecting the presence of hate speech and identifying further features in hateful contents, which included two sub-tasks. for subtask a, which was the hate speech (hs) category, the best performance was achieved by training a support vector machine (svm) model with a radial basis function (rbf) kernel. several other high scoring teams used a convolutional neural network (cnn) which was traditionally the most popular approach to this topic @xcite . for trac-1 @xcite , the challenge was to develop a classifier that could discriminate between three levels of aggression in social media. the results showed that with careful consideration, classifiers like svm and even random forest could perform at par with deep neural networks. however, in the end, more than half of the top 15 systems were trained on neural networks which demonstrates the approach's effectiveness. the introduction of bert @xcite ) spurred the use of pre-trained transformer models for classifying offensive speech @xcite . as a result, neural transformer based language models have increasingly become more popular in offensive language iden-tification. the use of pre-trained bert models, as well as bert-based models, was shown to be able to achieve competitive performance in popular competitions such as offenseval @xcite . language-specific and multilingual models have also been introduced to assist nlp research in various languages such as gbert for german @xcite , arabert for arabic @xcite , and the multilingual xlm-r @xcite that has been been applied to offensive language identification @xcite . ## data in the germeval-2021 dataset, the focus has been extended beyond the identification of offensive comments to include two additional classes: engaging comments that can motivate readers to participate in conversations, and fact-claiming comments. the dataset for this iteration of germeval comprises over 3,000 facebook user comments that have been extracted from the page of a political talk show of a german television broadcaster. the training dataset has a total of 3,244 instances and comprises 1,074 instances without any toxic, engaging or fact claiming content. in ## results we show the results for the evaluation set in the official leaderboard of the competition was not yet released at the time of writing this paper, therefore, after the organizers released the gold labels for the test set, we calculated the precision, recall, and f1 values for the test set. the results are shown in ## conclusion and future work in this paper, we presented the wlv-rit entry to germeval-2021. germeval-2021 provided participants with the opportunity of testing computational models to identify toxic, engaging, and fact claiming comments. we experimented with neural transformer models in stl environment and mtl environment. mtl environment consistently outperformed stl suggesting that the use of shared learning methods improves the performance of individual tasks. furthermore, we observed that pre-trained language-specific transformer models trained for german such as gbert and gelectra outperform mbert. finally, in addition to the transformer-based mtl approach, we could observe that the use of language modelling led performance improvement in some of the tasks. in the future, we would like to carry out an error analysis on the output of our systems to better understand the impact and limitations of mtl for these three tasks. finally, we would like to experiment with multi-task learning in other languages, particularly low-resource languages for which only limited language resources are available.