In Data We Trust: A Critical Analysis of Hate Speech Detection Datasets (2020)

Abstract: Recently, a few studies have discussed the limitations of datasets collected for the task of detecting hate speech from different viewpoints. We intend to contribute to the conversation by providing a consolidated overview of these issues pertaining to the data that debilitate research in this area. Specifically, we discuss how the varying pre-processing steps and the format for making data publicly available result in highly varying datasets that make an objective comparison between studies difficult and unfair. There is currently no study (to the best of our knowledge) focused on comparing the attributes of existing datasets for hate speech detection, outlining their limitations and recommending approaches for future research. This work intends to fill that gap and become the one-stop shop for information regarding hate speech datasets.

## introduction it is imperative to detect hateful speech on social media platforms and other online spaces because its real life implications are usually dire. the research community working towards achieving this goal spans from the social sciences to computer science. under the field of computer science, natural language processing (nlp) and machine learning (ml) techniques have been applied to this task of detecting hate speech by mostly framing it as a text classification task. here, text is classified into different categories based on its innate content or features. text classification is a supervised ml task; which means it requires a considerable amount of labelled data. each data instance needs a label or a class/category that it belongs to. although the majority of the studies in this research area use labelled data as they conduct a classification task, there are some that do not @xcite . in this study, we concentrate on datasets for hate speech detection in the english language while briefly highlighting other languages and similar concepts such as cyberbullying and abuse detection. the same issue discussed here are also true in other languages, thus all suggested solutions would persist. the overall aim of this work is to provide insight into the existing datasets and a consolidated analysis into their strengths and weaknesses and most importantly suggest methods to forward research in this area. to achieve this, we ask several questions: in its unbalanced form or not? although this work will be critiquing a few studies, it is not meant to be negative in any form. ## motivations the importance of hate speech detection research cannot be overemphasised. now, more than ever, with the current inflammatory political climate and discourse all around the world and minorities in various locations demanding for equality and equity, we cannot allow additional bias to be introduced into their lives through artificial intelligence. the problem of hate speech detection is one yet to be solved even to an acceptable level. it would be counter-productive if all the research efforts are not focused and channeled towards a better tomorrow by building on top one another. so we were motivated to go back to a root of the problem: the data. one of the foundations of this research work (that we can easily make changes on) is the data set. we can only build solid structures on solid foundations. furthermore, research efforts would be futile if the proposed state-of-the-art for this task fail to perform well on a realistic dataset. ## summary of existing datasets in this section, we highlight the currently existing datasets used in literature for the task of detecting hate speech. in the broad area of abusive language detection, there exists several other datasets collected and annotated for cyberbullying, toxicity, aggression and so on (we would not discuss those in-depth as they are out of the scope of this work). as highlighted in @xcite , the majority of the studies in this area of hate speech detection collected and annotated their own datasets, however, some were not made publicly available. the existing datasets are: of the annotated sample, 10.15% of sexual orientation category, 3.73% of race category, 2.66% of disability category and 11.68% of religion category are considered offensive or antagonistic. the dataset was collected after different trigger events for each category.