NTU _ NLP at S em E val-2020 Task 12: Identifying Offensive Tweets Using Hierarchical Multi-Task Learning Approach (2020)

Abstract: This paper presents our hierarchical multi-task learning (HMTL) and multi-task learning (MTL) approaches for improving the text encoder in Sub-tasks A, B, and C of Multilingual Offensive Language Identification in Social Media (SemEval-2020 Task 12). We show that using the MTL approach can greatly improve the performance of complex problems, i.e. Sub-tasks B and C. Coupled with a hierarchical approach, the performances are further improved. Overall, our best model, HMTL outperforms the baseline model by 3% and 2% of Macro F-score in Sub-tasks B and C of OffensEval 2020, respectively.

## introduction multilingual offensive language identification in social media (offenseval 2020) hosted by @xcite is a popular competition attracting many teams. the task is based on offensive language identification dataset (olid) 1.0 and a new dataset solid. offenseval 2020 consists of three subtasks. in sub-task a, the goal is to identify if a given tweet is offensive or non-offensive. in sub-task b, the focus is to identify if the offensive content in a tweet is targeted or untargeted. in sub-task c, systems have to detect the type of a target in an offensive tweet. the purpose of this paper is to investigate the multi-task learning (mtl) approach to the offenseval tasks. we propose the mtl model, which is based on the popular text encoder bert @xcite with the mtl architecture. to leverage the hierarchical nature of the three subtasks in offenseval 2020, we further propose the hmtl model, which is based on the bert with a hierarchical mtl architecture. this paper is organized as follows. we summarize the related work in section 2. section 3 introduces the dataset, the preprocessing procedure, and the architectures of our models. our results in offenseval 2019 and offenseval 2020 are reported and discussed in section 4. finally, section 5 concludes this paper. ## related work the overview paper of offenseval 2019 @xcite shows that 70% of the teams that participated last year use the deep learning models, and 8% of them use the bert model in sub-task a. many of the top-ranked teams use the bert model. the nuli team @xcite , which wins the first place in sub-task a, proposes a bert-based model and an lstm-based model. their work shows that the pre-trained bert model has a good competition result in the task. the vradivchev_anikolov team @xcite wins the first place and the second place in sub-tasks c and a, respectively. they explore several models, among which bert has the highest performance. the um-iu@ling team @xcite wins the third place in sub-task a. they use the pytorch framework to build the bert model in sub-task a, and use the svm model in sub-tasks b and c. multi-task learning (mtl) is one of widely-used strategies for improving neural network models. @xcite introduces the two most commonly used methods of mtl in deep learning, namely hard parameter sharing and soft parameter sharing approaches. @xcite propose a hierarchical multi-task (hmtl) model, by combining low-level simple tasks with high-level complex tasks to train. the model achieves the state-of-the-art results in named entity recognition, entity mention detection, and relation extraction. in this paper, we consider the hierarchical nature of the three tasks and introduces the hmtl framework to deal with the problems. ## results we show the results of our model in the offenseval 2019 and 2020 testsets in in comparing the results of offenseval 2019 and offenseval 2020, our observations are as follows. firstly, the performances of hmtl and mtl models in sub-task a are similar. this may be because sub-task a is relatively simple, the hierarchical approaches are not helpful. secondly, both hmtl and mtl outperform the vanilla bert model in all sub-tasks of offenseval 2019 and 2020, showing that the multi-task learning approach works very well in offenseval. thirdly, the performances of hmtl in sub-tasks b and c is almost higher than those of mtl model except that it is a little worse in offenseval 2019 sub-task c. this shows that the hierarchical method is useful in more complex problems, such as sub-tasks b and c. lastly, in terms of the average f-scores of each model in each subtask, hmtl is better than mtl, and mtl is better than bert. ## conclusion our multi-task learning approaches, mtl and hmtl, can effectively use the implicit information of sub-task a to improve the performance of sub-task b and sub-task c in the offenseval 2019 and 2020. for both offenseval evaluations, both mtl and hmtl models outperform the vanilla bert model in all subtasks. we show that the multi-task learning approach works very well in more complex problems. coupled with the hierarchical approach, performance can be further improved. our mtl and hmtl approaches can be applied to other classification tasks with a hierarchical nature. in the future, we will consider adding more complex structures after the bert layer of the hmtl model. we hope that low-level tasks can gain more knowledge from high-level tasks to improve the overall performance.