T he N orth at S em E val-2020 Task 12: Hate Speech Detection Using R o BERT a (2020)

Abstract: Hate speech detection on social media platforms is crucial as it helps to avoid severe situations, and severe harm to marginalized people and groups. The application of Natural Language Processing(NLP) and Deep Learning has garnered encouraging results in the task of hate speech detection. The expression of hate, however is varied and ever evolving. Thus, it is important for better detection systems to adapt to this variance. Because of this, researchers keep on collecting data and regularly come up with hate speech detection competitions. In this paper, we discuss our entry to one such competition, namely the English version of sub-task A for the OffensEval competition. Our contribution can be perceived through our results, which were first a F1-score of 0.9089, and with further refinements described here climb up to0.9166. It serves to give more support to our hypothesis that one of the variants of BERT (Devlin et al., 2018), namely RoBERTa can successfully differentiate between offensive and not-offensive tweets, given some preprocessing steps (also outlined here).

## introduction the present paper is a detailed description of the system foot_0 we use in the competition of detecting hate speech @xcite . the system proposed here was designed for the english language portion of sub-task a. we have decided to investigate a deep learning model, due to the considerable amount of data we had available at our disposal. the macro f1-score of 0.9087 attained by our model in the competition puts us close to the top half of the competitors. while this can be discouraging, it should be noted that only one million samples were used to train our models due to the limitations of the available hardware capacity. furthermore, even with the above-mentioned limitation, the difference between the classification score we attained, and the top team's classification score was only 0.02. this suggests that with some improvements (e.g. optimization of hyper-parameters and samples used), our system can be capable of state-of-the-art performance. ## experimental data regarding the data used, for training, we used the training data published for the competition (offense-val2020 @xcite ) -described in section 2.1), while for development and validation purposes, we used the olid dataset @xcite (described in section 2.2). as we aimed to detect hate speech for english, we used the english language data from both datasets. ## system overview in this section, we give an overview of the full processing chain created to produce the results that had been submitted for the competition. first, we describe the text preprocessing steps we took to clean up the tweets and make them more usable to the whole system. then, the focus will be on the selection of the model from the two models that were decided to test (distilbert @xcite and roberta @xcite ), as well as, the reasons that lead us to select the final model (in this case roberta @xcite ). we will also discuss the hyper-parameters that gave us our (previously indicated) best score. lastly, we discuss the algorithm we used once the output of our model was collected to transform it from a regression task to a classification one. ## experimental setup for training the model, the data was offenseeval2020 data @xcite raw was put to the preprocessing stage described previously. after we had the clean data, we took the first 1 million samples from the training data, and partitioned this one million tweets into a dev set of 3000 tweets (constructed from the last 3000 tweets), and the rest was used for training. the resulting training set was fed as input to the roberta @xcite model. the hyper-parameters were mostly left as default, though we modified the following: "early stopping patience" this was set to three and "learning rate" was set to 1e -5 and "evaluate during training steps", the number of training steps before evaluating, was set to 2000. with these hyper-parameters, we trained two versions of distilbert (one for 3 epochs, and another one for 20 epochs), as well as roberta, for 3 epochs. here, we used only the mean (and not the standard deviation) score as target for regression. as the final step of the process, predicted scores had to be transformed into class labels. to do so, we set a threshold for each model on the olid train data. this threshold was selected for each model based on the best result attainable on the training set of the olid database. this threshold and the corresponding result for each model are listed in ## quantitative analysis here we describe what hyper-parameter changes (we only varied two) or modeling (also includes the two) decisions we think affected our results. as presented in ## conclusion to conclude this paper, our system shows a good performance while not being overly complicated; we show that it is possible to tackle the task of hate speech detection using one of the variants of bert @xcite , in this case, roberta @xcite . we are obtaining an f1-score of 0.9087 (final result), which can be further pushed down to content moderators. for future work, we will continue focusing on the model proposed here with slight modifications to hyper-parameters and text preprocessing. we think that getting a 0.92 is quite possible with further optimization of hyper-parameters, and by increasing the size of the development set. we also plan to experience with ensembles of models trained on different portions of the train set.