Lee at S em E val-2020 Task 12: A BERT Model Based on the Maximum Self-ensemble Strategy for Identifying Offensive Language (2020)

Abstract: This article describes the system submitted to SemEval 2020 Task 12: OffensEval 2020. This task aims to identify and classify offensive languages in different languages on social media. We only participate in the English part of subtask A, which aims to identify offensive languages in English. To solve this task, we propose a BERT model system based on the transform mechanism, and use the maximum self-ensemble to improve model performance. Our model achieved a macro F1 score of 0.913(ranked 13/82) in subtask A.

## introduction in recent years, due to the rapid development of mobile internet and social media platforms, people have begun to use a variety of social media to share their lives, such as facebook and twitter. people share their something like events on social media, and this behavior can receive good or bad comments. some bad reviews slowly evolve into offensive language. a large amount of offensive language is flooded in social media, and filtering out offensive language has become an important thing. because manual screening is very time consuming and can cause symptoms such as post-traumatic stress disorder, many studies have focused on automating this process. task 12 of semeval 2020 @xcite is the second edition of task 6 of semeval 2019 @xcite . the purpose of this task is to identify online offensive language . its goal is to use computational methods to identify offensive and hate speech in user-generated content on online social media platforms. we can use this method to prevent the abuse of offensive words in social media. this task gives us some social media platforms and categorizes the content in this task, we only participate in subtask a @xcite : recognizing offensive language. for this task, we use deep learning methods. we mainly use bert model based on transformer mechanism and model ensemble. after data processing, we input the generated word vector into the pre-trained model. in addition, we also adopted the method of model ensemble to optimize the prediction probability of the results in order to get better results. facts have proved that this approach is an effective method. the rest of our paper is structured as follows. section 2 describes data preparation. models are described in section 3. experiments and evaluation are described in section 4. the conclusions are drawn in section 5. ## data preparation the organizers provided training and test sets @xcite , containing 9073396 and 3887 sentences respectively. we note that the data set format of task 12 is different from that of the previous year @xcite . therefore, before we perform data preprocessing, we need to process the data format. the data format of the obtained task 12 is shown in figure 1 . we did not get a clear label data. we get the data about the label including av g and st d. where av g is the average confidence of a particular instance predicted by several supervised models. st d is figure 1 : an example from the semeval 2020 task 12 dataset the standard deviation of the confidence of a particular instance from av g. in order to get the label, we need to calculate the confidence of the models to get the value of the label. for the characteristics of the data format, we have two calculation methods to calculate the confidence of the data. the calculation formula is shown below. since we only participated in subtask a, and this is a binary task. therefore, we define the sentence with the calculated confidence greater than 0.50 as the label 1. where the label 1 represents offensive language, on the other hand, the label 0 represents non-aggressive language. the processed data format is shown in figure 2 . after modifying the data format, we can start data preprocessing. the text from the tweets is inherently heavily useless. tweets are processed using the tweettokenize tool @xcite . cleaning the text before further processing helps to generate better functionality and semantics. we perform the following preprocessing steps. ## models after data preprocessing, we can start training the model. in this task, the model that we use is the bert model based on the transform mechanism. in order to improve the effect of the model, we also adopted the model ensemble method to improve the model effect. facts have proved that this has indeed been improved. the model is shown in figure 2 . in the following, we will introduce the details of the model we use. figure 3 : the architecture of the bert model in our task ## experiments and evaluation in this task, we use the bert model to train the task. for the bert model, the main parameters we focused on are the training step size, batch size, epoch, and learning rate. after learning the parameter adjustment for similar tasks, we fine-tune the model parameters. our parameter fine-tuning is shown in ## conclusion in this task, we mainly use bert model based on transformer mechanism and model ensemble. after testing, we found that the performance of the single model is slightly worse than the ensemble model. and there are some differences between the results of different parameters of the same model. our results are still not as satisfactory as the top teams on the leaderboard. in the future, we will continue to adjust the model, improve the hardware configuration of the computer, collect more external data, and conduct more experiments to obtain better results. meanwhile, we will try to solve the problem of data imbalance. we will discuss the impact of data preprocessing on model performance. we will continue to carry out model optimization and try more ensemble model methods.