Harmful Language Datasets: An Assessment of Robustness (2023)

Abstract: The automated detection of harmful language has been of great importance for the online world, especially with the growing importance of social media and, consequently, polarisation. There are many open challenges to high quality detection of harmful text, from dataset creation to generalisable application, thus calling for more systematic studies. In this paper, we explore re-annotation as a means of examining the robustness of already existing labelled datasets, showing that, despite using alternative definitions, the inter-annotator agreement remains very inconsistent, highlighting the intrinsically subjective and variable nature of the task. In addition, we build automatic toxicity detectors using the existing datasets, with their original labels, and we evaluate them on our multi-definition and multi-source datasets. Surprisingly, while other studies show that hate speech detection models perform better on data that are derived from the same distribution as the training set, our analysis demonstrates this is not necessarily true.

## introduction many forms of harmful language impact social media despite efforts -legal and technologicalto suppress it. 1 social media has been under significant scrutiny with regard to the effectiveness of their anti-hate speech policies, which usually involve users manually reporting a potentially malicious post in order to trigger a human review, and platforms adjusting their community guidelines by, for example, banning hateful comments, and employing automated moderation assistants. a robust and general solution to the problem does not yet exist, and given that there are many factors that influence the phenomenon of online hate speech, we expect this area of research to continue to pose significant challenges. one of the main reasons is that harmful language detection is an inherently subjective task. there have been many attempts to approach harmful language detection by introducing or selecting specific definitions @xcite . from blanket terms, such as abusiveness and offensiveness to sub-categories, such as misogyny and cyber-bullying, researchers have explored many variants. however, this begs the question of how to select and compare the possible definitions, especially when some categories are more efficient for cross-dataset training than others @xcite . the problem gets more intricate when multiple languages are involved, and when the translation of a term does not necessarily carry the same implications as in the source language. this can have significant implications for the development of cross-lingual systems @xcite . in this study, we attempt to shed light on the effectiveness of different definitions of harmful language both for annotation purposes and model development. we use the term "harmful language" as a wildcard term that can be potentially replaced with terms like toxic, hate speech, and offensiveness, among others. we perform a re-annotation of existing datasets with a range of definitions and replicate the experiments to assess robustness. then, we perform a qualitative error analysis on the re-annotations, showing that even instances that contain potentially harmful terms might not be perceived as harmful by annotators, underlining the subjectivity of the task. finally, we analyse the generalisability of the existing datasets across the different definitions by training bert-based classifiers with the original annotations and with our reannotations, concluding that evaluating on broader definitions can yield higher accuracy. the rest of this article is structured as follows. section 2 overviews existing studies on the issue of the definition of harmful language and its implications, as well as how state-of-the-art (sota) sys-tems handle generalisability. section 3 presents our re-annotation strategy. in section 4, we describe our experimental setup for training and evaluating with the the original and the re-annotated datasets. finally, after presenting our results in section 4.3, we assess our contribution in section 5, concluding by speculating on limitations and future work. disclaimer: this paper contains potentially offensive, toxic, or otherwise harmful language. ## related work harmful language is becoming all the more frequent due to the widespread use of social media and the internet, thus creating a vicious cycle that compromises the civility of the online community and threatens a healthy user experience @xcite . the need for automatically moderating toxic language has led to the development of a considerable body of related work, proposing solutions and highlighting existing problems. ## methodology our methodology is divided in two parts. the first part investigates whether closely-related definitions have an effect on inter-annotator agreement while the second part examines the compatibility and versatility of the present datasets by using them to train models. ## discussion taking into account the existing literature @xcite @xcite , this study confirms that models face a serious difficulty generalising. yet, our results show a promising aspect when it comes to model reproducibility for harmful language detection purposes, as well as building robust datasets though a robust annotation procedure. ## conclusion in spite of recent advances, model generalisation and method robustness still has a long way to go especially regarding harmful language online. in this study, we attempt to shed some light on the issue, first, by performing a re-annotation experiment with existing datasets employing crowdsourcing annotators and, second, by using the same datasets to train a baseline model as an automatic annotator. the human annotation shows that, although in most cases the annotations were inconsistent, toxicity and hota (any of the following: hateful, offensive, toxic, abusive language) appear to be the most consistent definitions, indicating that the broader the term used the more robust the annotations the experimental model, on the other hand, showed that, assessing on data from the same source as the training set, when using the original ground truth, can yield higher accuracy compared to assessing data from a different source, confirming previous studies. yet, this cannot be used as a rule of thumb since testing on the re-annotations showed that the performance can drop when testing on the data from the same source as the training set and it can increase when testing on previously completely unseen data.