Nova-Wang at S em E val-2020 Task 12: O ffens E mblert: An Ensemble of O ffensive Language Classifiers (2020)

Abstract: This paper presents our contribution to the Offensive Language Classification Task (English SubTask A) of Semeval 2020. We propose different Bert models trained on several offensive language classification and profanity datasets, and combine their output predictions in an ensemble model. We experimented with different ensemble approaches, such as SVMs, Gradient boosting, AdaBoosting and Logistic Regression. We further propose an under-sampling approach of the current SOLID dataset, which removed the most uncertain partitions of the dataset, increasing the recall of the dataset. Our best model, an average ensemble of four different Bert models, achieved 11th place out of 82 participants with a macro F1 score of 0.91344 in the English SubTask A.

## introduction as people turn more and more to the online world for their entertainment, social and communication needs, the anonymity offered by the platforms also draw out diverse and sometimes divisive opinions that can lead to abuse, bullying and mental stress. to ensure civility of online discussions while allowing for opposing ideas to be expressed, we need an effective way to detect offensive language in social media. the semeval 2020 task 12: "offenseval 2: multilingual offensive language identification in social media" @xcite , introduces a multi-lingual dataset with 5 languages (arabic, danish, greek, turkish and english). it is a continuation and extension of semeval 2019 task 6: "offenseval: identifying and categorizing offensive language in social media" @xcite . this paper presents the model submitted to the english stream of subtask a, focusing on the offensive language classification, where for a given tweet we classify it as offensive (off) if it contains any form of profanity or targeted offense, either veiled or direct, and non-offensive (non) otherwise. the official offenseval 2020 english training dataset @xcite was labelled using a semi-supervised method with several different models. the dataset provides a score which corresponds to an average prediction confidence of being offensive over all the models (µ) along with the uncertainty of the prediction given by their standard deviation (σ). the main challenge of using this dataset concerns with how to make an effective use of this large dataset as the tweets with mid-range average confidence values (µ) often have high variance (σ) as a result are imprecise and difficult to interpret. to overcome this challenge, we created two sub-sampled datasets @xcite , by removing majority of mid-range score µ (uncertain predictions) and with high variance σ from both sets, and by evenly sample from positive and negative values on the second set (set b). we also incorporated additional datasets into our model to balance out the uncertainty of the semisupervised set. rather than creating one single combined dataset, we trained separate models for each different dataset and experimented with various ensemble techniques. this allowed for more flexibility in tuning the impact of each dataset and allowed us to get the benefit of larger datasets without undermining smaller datasets. we fine-tuned a bert model for each dataset. we chose this model architecture as it has been shown to outperform models built using other structures for offensive language tasks. @xcite ; 2 related work the abusive and offensive language classification task has been studied from different perspectives, ranging from aggression identification @xcite , bullying detection @xcite , hate speech identification @xcite , toxic comment classification @xcite , and offensive language classification @xcite . last year edition of offenseval 2019 proposed a three-level hierarchical schema for offensive language classification @xcite . this dataset considers whether (i) a tweet is offensive or not, (ii) whether the offense is targeted, and (iii) whether it is targeted towards individuals, groups or others. the associated tasks were divided into three corresponding sub-tasks. for the first subtask, offensive language classification, models with bert @xcite architecture consistently out-performed other methods and were used by seven of the top ten teams. we followed a similar method to the top teams, @xcite and @xcite , using their recommended steps for pre-processing the tweets and fine-tuning pre-trained bert model. in a related kaggle competition in 2019, jigsaw published a dataset for identifying toxicity and minimising bias in online comments @xcite . the winning model was a blend of 2x xlnet, 2x bert and gpt2 medium @xcite . we did not follow the approach due to time and resource constraints and instead opted to use the dataset to supplement our existing data. @xcite proposed a separate two-fold topology for synthesizing different subtasks in abusive language detection by considering whether (i) the abuse is directed at a specific target and (ii) the degree to which it is explicit. they noted that implicit abuse is more difficult to identify, sometimes requiring more detailed annotation guidelines and perhaps even expert annotators. ## offenseval proposed system we trained several models using different datasets and combined the best ones in an ensemble. in this section we describe the datasets that were considered and how the models were trained. ## experimental results we created train/dev splits with the sizes specified in contrary to our initial hypothesis, the models trained from under-sampling the uncertain partition of solid did not perform as well as the model built using the whole set. we posit that further tuning of the optimal threshold µ and σ for under-sampling, may influence the results. possibly apply a more aggressive under-sampling could increase precision scores. due to time constraints we leave further exploration as future work. seeing that each model has different strengths, we further experiment with various ensemble techniques and combine these four models -olid, solid, kaggle and profanity. we reported the best model on the dev set (olid test). the average weighted ensemble (avg.) used equal weight for each of the four component models. for the grid search ensemble (grid), we searched weights from 0 to 1, in steps of 0.1. this resulted in over 1000 combinations with the same best results, possibly over-fitting due to the small dev set size ( 860 examples). for the remainder of the ensembles, we used 10 fold cross validation combined with grid search on the hyper-parameters to find the best parameters for the ensembles. we report the model ensemble results in ## conclusions and future work there are offensive language classification datasets that portray different annotation guidelines and different annotators' subjectivity. in this world we take advantage of the heterogeneous nature of these datasets to improve the performance of offensive language identification, by combining several models in an ensemble. this showed the importance of having training data that is reliable and diverse enough to capture different types of scenarios, and the potential benefit of combining and consolidating those datasets. we hypothesised that under-sampling uncertain and possibly mis-classified tweets we could improve the performance of classification algorithms, but the results so far have proven to be inconclusive. we leave as future work more aggressive under-sampling schemes to access the consequences of only training on highly confident predictions, and differ additional semi-supervised strategies to improve results using solid dataset. we show that using different sources for training offensive language classification tasks helps improve the quality of the predictions.