Kungfupanda at S em E val-2020 Task 12: BERT -Based Multi- T ask L earning for Offensive Language Detection (2020)

Abstract: Nowadays, offensive content in social media has become a serious problem, and automatically detecting offensive language is an essential task. In this paper, we build an offensive language detection system, which combines multi-task learning with BERT-based models. Using a pre-trained language model such as BERT, we can effectively learn the representations for noisy text in social media. Besides, to boost the performance of offensive language detection, we leverage the supervision signals from other related tasks. In the OffensEval-2020 competition, our model achieves 91.51% F1 score in English Sub-task A, which is comparable to the first place (92.23%F1). An empirical analysis is provided to explain the effectiveness of our approaches.

## introduction nowadays, offensive content has invaded social media and becomes a serious problem for government organizations, online communities, and social media platforms. therefore, it is essential to automatically detect and throttle the offensive content before it appears in social media. previous studies have investigated different aspects of offensive languages such as abusive language @xcite and hate speech @xcite . recently, @xcite first studied the target of the offensive language in twitter and @xcite expand it into the multilingual version, which is practical for studying hate speech concerning a specific target. the task is based on a three-level hierarchical annotation schema that encompasses the following three general sub-tasks: (a) offensive language detection; (b) categorization of offensive language; (c) offensive language target identification. to tackle this problem, we emphasize that it is crucial to leverage pre-trained language model (e.g., bert @xcite ) to better understand the meaning of sentences and generate expressive wordlevel representations due to the inherent data noise (e.g., misspelling, grammatical mistakes) in social media (e.g., twitter). in addition, we hypothesize that the internal connections exist among the three general sub-tasks, and to improve one task, we can leverage the information of the other two tasks. therefore, we first generate the representations of the input text based on the pre-trained language model bert, and then we conduct multi-task learning based on the representations. experimental results show that leveraging more task information can improve the offensive language detection performance. in the offenseval-2020 competition, our system achieves 91.51% macro-f1 score in english sub-task a (ranked 7th out of 85 submissions). especially, only the olid @xcite is used to train our model and no additional data is used. our code is available at: https: //github.com/wenliangdai/multi-task-offensive-language-detection . ## related works in general, offensive language detection includes some particular types, such as aggression identification @xcite , bullying detection @xcite and hate speech identification @xcite . @xcite applied concepts from nlp to exploit the lexical syntactic feature of sentences for offensive language detection. @xcite integrated the textual features with social network features, which significantly improved cyberbullying detection. @xcite and @xcite used convolutional neural network in the hate-speech detection in tweets. recently, @xcite introduce an offensive language identification dataset, which aims to detect the type and the target of offensive posts in social media. @xcite expanded this dataset into the multilingual version, which advances the multilingual research in this area. pre-trained language models, such as elmo @xcite and bert @xcite have achieved great performance on a variety of tasks. many recent papers have used a basic recipe of finetuning such pre-trained models on a certain domain @xcite @xcite or on downstream tasks @xcite @xcite . ## datasets in this project, two datasets are involved, which are the dataset of offenseval-2019 and offenseval-2020 respectively. in this section, we introduce the details of them and discuss our data pre-processing methods. ## methodology we propose a multi-task learning (mtl) method (figure 1(b) ) for this offensive language detection task. it takes good advantage of the nature of the olid @xcite , and achieves an excellent result comparable to state-of-the-art performance only with the olid @xcite and no external data resources. a thorough analysis is provided in section 5.2 to explain the reasons of not using the new solid dataset created in offenseval-2020 @xcite . ## experiments during the training phase, we evaluate our models on the test set of olid (offenseval-2019). as a reference, we also evaluate them on the test set of solid (offenseval-2020), which is only released after the submission date. ## conclusion and future work from all of our experiments, we conclude that mtl improves the performance of sub-task a in both olid and solid. moreover, our finding shows that pre-training sub-task a as a regression task doesn't improve the model's performance. we think that there are several paths for further work. firstly, more studies about the combination of the sub-tasks can be investigated for mtl. this can show us more about the interaction between sub-tasks, and how much does one influence another. secondly, as mentioned in @xcite , simultaneously updating the model's parameters during mtl can have negative effects on optimization as the total gradients are too noisy. this becomes more significant when the number of tasks is large or the batch size is small. as a result, asynchronous optimizations for each task may provide a more stable gradient descent.