KAFK at S em E val-2020 Task 12: Checkpoint Ensemble of Transformers for Hate Speech Classification (2020)

Abstract: This paper presents the approach of Team KAFK for the English edition of SemEval-2020 Task 12. We use checkpoint ensembling to create ensembles of BERT-based transformers and show that it can improve the performance of classification systems. We explore attention mask dropout to mitigate for the poor constructs of social media texts. Our classifiers scored macro-f1 of 0.909, 0.551 and 0.616 for subtasks A, B and C respectively. The code is publicly released online.

## introduction the research community has put much effort over the last few years in developing automated detection methods to combat hate speech in social media @xcite . but the complex nature of this phenomenon shows that it has no single solution. the second edition of the offenseval workshop @xcite , titled offenseval-2020, is organized with the goal of promoting further research in this domain. the offenseval-2020 workshop features shared subtasks in five different languages. we participated in the english language which consists of three subtasks @xcite . each subtask is a breakdown of the taxonomy of offensive content. subtask a (offensive language identification) is the classification of a post as offensive @xcite or not offensive @xcite . subtask b (automatic categorization of offence types) determines whether a post containing an insult or a threat is targeted towards an individual, a group, or other @xcite or simply contains non-targeted profanity and swearing @xcite . and subtask c (offense target identification) is the identification of the target of an offensive post. the targets being individual @xcite , group [grp] and other @xcite . ## data the dataset for the english edition of semeval task-12 is compiled by @xcite . it is a large scale dataset of loosely labelled text samples, i.e, each sample is associated with an average confidence measure and it's standard deviation. average confidence is the average of the confidences with which numerous supervised models predicted an instance as belonging to the positive class for a subtask. the positive class is off for subtask a, and unt for subtask b. the dataset of subtask c has no positive class, instead, the average confidence of each class is given. in our study, we assign an instance to a class if its average confidence for that class is greater than or equal to 0.50. the label counts for each subtask is given in figure 1 . 4 methodology 4.1 checkpoint ensembling model ensembling refers to the method of constructing a single classifier from a collection of different classifiers @xcite . however, creating ensembles of large deep neural networks (dnns), like the ones used in this study, is an expensive task. it is generally not possible to train multiple models with limited time and gpu resources. hence, based on the work by @xcite , we used a simple checkpoint ensembling method for creating the transformer-based ensemble classifiers used in this study. in checkpoint ensembling, a copy of the model is saved at each checkpoint. these copies are later combined in some fashion to make the classification (see figure 2a ). in contrast, a traditional ensemble usually combines different models (see figure 2b ). in our checkpoint ensembling approach, we save the dev set predictions and weights of the dnn models at each epoch and apply algorithm 1 to determine which models to use for the ensemble. algorithm 1 uses the dev set predictions to create a list of models to use in the ensemble. algorithm 1 picks a model if using its predictions improves the metric (macro-f1 for offenseval), otherwise not. algorithm 1 is called twice with reverse set to t rue and then f alse. if the ensemble doesn't improve the metric, we can simply choose the best model found during training. after determining the models, we apply algorithm 2 to get the final predictions. algorithm 2 simply adds the predictions of the classifying layer of the chosen models and uses argmax along each row to get the final prediction. ## results and error analysis ensembled roberta for subtask b performed poorly. it failed to distinguish properly between targeted and untargeted samples. untargeted samples such as • "@user yeah my deck was insane that run, but normally i suck lol " were predicted as targeted. we found that samples which contained the "@user" token were mostly mistaken as targeted. the confusion matrix is given in figure 4b .