PUM at S em E val-2020 Task 12: Aggregation of Transformer-based Models’ Features for Offensive Language Recognition (2020)

Abstract: In this paper, we describe the PUM team’s entry to the SemEval-2020 Task 12. Creating our solution involved leveraging two well-known pretrained models used in natural language processing: BERT and XLNet, which achieve state-of-the-art results in multiple NLP tasks. The models were fine-tuned for each subtask separately and features taken from their hidden layers were combinedand fed into a fully connected neural network. The model using aggregated Transformer featurescan serve as a powerful tool for offensive language identification problem. Our team was ranked7th out of 40 in Sub-task C - Offense target identification with 64.727% macro F1-score and 64thout of 85 in Sub-task A - Offensive language identification (89.726% F1-score).

## introduction nowadays, abusive content spreading in social media poses a serious problem. the offensive language is pervasive and due to the users' anonymity on the internet, people feel unreachable when it comes to the punishment. as a result, many people may suffer from being insulted. multiple social media platforms and companies noticed the emerging problem and have already investigated possible solutions. one of them is to use computational methods to detect hate speech in various types of utterances posted on the web, what has already drawn considerable attention as evidenced in recent publications @xcite , @xcite , @xcite . nevertheless, the ability of identifying offense automatically is the first step which has to be undertaken. in this paper we tackle the problem of identifying and categorizing offensive language in social media. it is the 12-th task of semeval-2020 competition @xcite . this is also the second edition of offenseval, which started in 2019 @xcite . the problem is to detect and assign offensive language occurrences to various pre-defined categories using natural language processing methods. a dedicated datasets consisting of tweets crawled from twitter were created specifically for this task. organizers of the competition provided the participants also with the dataset from previous year's edition -olid @xcite ), which we used during the development of our solution. each example is appropriately labelled according to a specific subtask: offenseval 2020 features a multilingual dataset with five languages. the languages included are: arabic, danish, english, greek and turkish @xcite , (sigurbergsson and derczynski, 2020), @xcite , @xcite . from these 5 languages the team members know only english, thus we decided to limit the consideration to this language. this work is licensed under a creative commons attribution 4.0 international licence. licence details: http: //creativecommons.org/licenses/by/4.0/ . since training dataset from this year consists of examples with confidence measures produced by unsupervised learning methods instead of labels, we decided to map the scores to the labels. in order to achieve that we simply apply the threshold of 0.4 to the avg conf value in the training phase. the number of tweets in training and test datasets for subtasks in which we participated are presented in moreover, we had to tackle also a significant problem in both subtasks namely, class imbalance. for example taking into account provided olid dataset, as for subtask a, it contained 8840 examples labelled as not offensive, whereas only 4400 examples were offensive (proportion 2:1). as for subtask c, the imbalance was even more significant, 2407 examples were labelled as individual offensiveness target, 1074 as group target and only 395 as other (proportion 6:3:1). ## methodology for development of our solution we used olid dataset. 90% of the dataset served for training models and the rest was used as validation set for choosing the best hyperparameters. as for the evaluation of our models' performance we used test dataset from the last year's edition of offenseval (with 860 examples). ## experiments in order to check the performance of designed model and compare it to the baselines, we conducted the experiments presented in the section below. we used the test set from last year's competition edition for subtask a, as it also consisted of tweets and concerned recognizing if an utterance was offensive or not.