Hitachi at S em E val-2020 Task 12: Offensive Language Identification with Noisy Labels Using Statistical Sampling and Post-Processing (2020)

Abstract: In this paper, we present our participation in SemEval-2020 Task-12 Subtask-A (English Language) which focuses on offensive language identification from noisy labels. To this end, we developed a hybrid system with the BERT classifier trained with tweets selected using Statistical Sampling Algorithm (SA) and Post-Processed (PP) using an offensive wordlist. Our developed system achieved 34th position with Macro-averaged F1-score (Macro-F1) of 0.90913 over both offensive and non-offensive classes. We further show comprehensive results and error analysis to assist future research in offensive language identification with noisy labels.

## introduction offensive language identification focuses on identifying abusive languages, such as threats, insults, calumniation, discrimination, swearing @xcite which could be targeted at an individual or group. typically, offensive language identification is helpful for (i) content moderators who face an increasing volume of abusive content and would like assistance in managing it efficiently (ii) social media sites which do auto-censoring of violence and hate speech and (iii) search query optimization (iv) improving conversations across messenger platforms @xcite . in recent times offensive language identification has seen a range of works including the creation of a large number of the corpus, open challenges, and benchmarking competitions. moreover, with recent advances in the area of language understanding and transfer learning in nlp, offensive language identification has seen large traction @xcite . while all these have significantly improved the capacity to identify offensive languages, there are some challenges that require addressing namely (i) noisy labels as a result of task formulation of semeval-2020 task 12 subtask-a (english language) which focuses on the identification of offensive language with noisy labels to mitigate the problem of need of large dataset @xcite ) and (ii) identification rare and idiosyncratic offensive words @xcite . in this paper, we aim to address this task by developing a hybrid system with two major contributions, respectively solving previously mentioned challenges. as a first contribution, we introduce a statistical sampling algorithm that selects trustworthy tweets using statistics of noisy labels and auxiliary datasets. while sampling datasets to reduce noise has been widely explored @xcite , to the best of our knowledge ours is the very first work in the context of offensive language identification. as a second contribution, we concentrate on the post-processing method that uses words from an offensive wordlist to identify tweets with rare and idiosyncratic offensive words. with these, our proposed system achieved 34 th place with macro-averaged f1-score (macro-f1) of 0.90913 over both offensive and non-offensive classes. the rest of the paper is organized as follows. in section 2, we discuss literature on text classification with noisy labels. in section 3, we present the dataset of all the subtasks, followed by a detailed system description in section 4, with a focus on the statistical sampling algorithm, data splits, hyperparameters, and the post-processing approach. in section 5, we describe our results, error analysis, and some of our findings foot_0 . finally, in section 6, we conclude with a summary and possible suggestions for future work. ## related work text classification using noisy labels in the general domain has seen works over two-decade which focuses on learning from noisy labels for a wide range of classifiers including svm @xcite and fisher discriminant @xcite . while traditional predominantly approaches handle label noise by detecting and eliminating the corrupted labels, recent deep learning approaches either reduce the effect of noisy labels by using auxiliary clean dataset @xcite @xcite or attempt to learn the noise distribution @xcite @xcite . despite a plethora of works in general domain, text classification using noisy labels is still in a nascent stage @xcite . in this work, we blend the best of both traditional @xcite and deep learning approaches @xcite , where we introduce a simple statistical sampling algorithm, which detects trustworthy labels using the standard deviation of confidence and an auxiliary clean dataset. to the best of our knowledge, this is the first work that uses sampling datasets to handle noisy labels in the context of offensive language identification. ## dataset the dataset for semeval-2020 task-12 (english language) offensive language identification in social media that is made available to participants contains a total of 9,275,847 noisy labeled tweets @xcite ) distributed across the three subtasks (see subtask-a subtask-b subtask-c number of tweets 9,087,119 1,88,728 1,13,803 ## system description as mentioned earlier, for this task we developed a hybrid system with three major components namely (i) statistical sampling algorithm (sa) that selects trustworthy tweets for training with noisy labels (ii) classifier which uses a bert @xcite language model fine-tuned for classification and (iii) post-processing (pp) component which uses words from wordlist to override the potentially false-negative predictions from the bert model. each of the three components is described in sections 4.1-4.3. ## conclusion offensive language identification is still a challenging task with multiple challenges including the need of learning from noisy labels due to lack of availability of large datasets and difficulty in identification of idiosyncratic/rare offensive words. in this work, to address these issues we developed a hybrid system with the bert classifier trained with tweets selected using statistical sampling algorithm (sa) (section 4.1) and post-processing (pp) (section 4.3) using an offensive wordlist. the sampling algorithm lessens the impact of the noisy label by selecting trustworthy samples, meanwhile, wordlist helps the identification of idiosyncratic and rare offensive words. with these, our final submitted system achieved a macro-f1 of 0.90913 in the leader board. further, in section 5.1 we presented detailed results to find that the proposed system did not produce any false negatives. besides, we saw that the developed system produces absolute recall for offensive tweets. at the same time, we also saw that post-processing indeed causes a drop in performance due to an increase in false positives and warranted more detailed study. finally, in section 5.2 we detailed various errors to identify rhetorical statements, the presence of swear words as the major contributing factors, humor, and rare words are found to be minor contributors for the errors. additionally, we also saw the issues of lack of context, lack of guidelines for offensive tweet annotation visible through labeling ambiguity for some of the tweets. in future works, in addition to addressing previously mentioned problems, we will also focus on extensive analysis of the effect of hyperparameters on results across multiple datasets.