Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021)

Abstract: We present a human-and-model-in-the-loop process for dynamically generating datasets and training better performing and more robust hate detection models. We provide a new dataset of 40,000 entries, generated and labelled by trained annotators over four rounds of dynamic data creation. It includes 15,000 challenging perturbations and each hateful entry has fine-grained labels for the type and target of hate. Hateful entries make up 54% of the dataset, which is substantially higher than comparable datasets. We show that model performance is substantially improved using this approach. Models trained on later rounds of data collection perform better on test sets and are harder for annotators to trick. They also have better performance on HateCheck, a suite of functional tests for online hate detection. We provide the code, dataset and annotation guidelines for other researchers to use.

## introduction accurate detection of online hate speech is important for ensuring that such content can be found and tackled scalably, minimizing the risk that harm will be inflicted on victims and making online spaces more accessible and safe. however, detecting online hate has proven remarkably difficult and concerns have been raised about the performance, robustness, generalisability and fairness of even stateof-the-art models @xcite @xcite @xcite . to address these challenges, we present a human-and-model-in-the-loop process for collecting data and training hate detection models. our approach encompasses four rounds of data generation and model training. we first trained a classification model using previously released hate speech datasets. we then tasked annotators with presenting content that would trick the model and yield misclassifications. at the end of the round we trained a new model using the newly presented data. in the next round the process was repeated with the new model in the loop for the annotators to trick. we had four rounds but this approach could, in principle, be continued indefinitely. round 1 contains original content created synthetically by annotators. rounds 2, 3 and 4 are split into half original content and half perturbations. the perturbations are challenging 'contrast sets', which manipulate the original text just enough to flip the label (e.g. from 'hate' to 'not hate') @xcite . in rounds 3 and 4 we also tasked annotators with exploring specific types of hate and taking close inspiration from real-world hate sites to make content as adversarial, realistic, and varied as possible. models have lower accuracy when evaluated on test sets from later rounds as the content becomes more adversarial. similarly, the rate at which annotators trick models also decreases as rounds progress (see we make three contributions to online hate classification research. first, we present a human-andmodel-in-the-loop process for training online hate detection models. second, we present a dataset of 40, 000 entries, of which 54% are hate. it includes fine-grained annotations by trained annotators for label, type and target (where applicable). third, we present high quality and robust hate detection models. all data, code and annotation guidelines are available. 1 2 background benchmark datasets several benchmark datasets have been put forward for online hate classification @xcite @xcite @xcite @xcite @xcite . these datasets offer a comparison point for detection systems and have focused the field's attention on important subtasks, such as classification across different languages, domains and targets of hate. performance on some benchmark datasets has increased substantially through the use of more advanced models. for instance, in the original @xcite paper in 2016, the authors achieved an f1 of 0.74. by 2018 this had increased to 0.93 @xcite . numerous problems have been identified with hate speech training datasets, such as lacking linguistic variety, being inexpertly annotated and degrading over time @xcite . @xcite examined 63 open-source abusive language datasets and found that 27 (43%) were sourced from twitter @xcite . in addition, many datasets are formed with bootstrapped sampling, such as keyword searches, due to the low prevalence of hate speech 'in the wild' @xcite . such bootstrapping can substantially bias the nature and coverage of datasets @xcite . models trained on historical data may also not be effective for present-day hate classification models given how quickly online conversations evolve @xcite . model limitations systems trained on existing datasets have been shown to lack accuracy, robustness and generalisability, creating a range of false positives and false negatives @xcite @xcite @xcite . these errors often make models unsuitable for use in downstream tasks, such as moderating online content or measuring online hate. 2019a). ## dataset labels previous research shows the limitations of using only a binary labelling schema (i.e., 'hate' and 'not hate'). however, there are few established taxonomies and standards in online hate research, and most of the existing datasets have been labelled with very different schemas. the hierarchical taxonomy we present aims for a balance between granularity versus conceptual distinctiveness and annotation simplicity, following the guidance of @xcite . all entries are assigned to either 'hate' or 'not hate'. 'hate' is defined as "abusive speech targeting specific group characteristics, such as ethnic origin, religion, gender, or sexual orientation." @xcite . for 'hate', we also annotate secondary labels for the type and target of hate. the taxonomy for the type of hate draws on and extends previous work, including @xcite hovy (2016); vidgen et al. (2019a); zampieri et al. (2019) . ## annotation data was annotated using an open-source web platform for dynamic dataset creation and model benchmarking. 3 the platform supports human-andmodel-in-the-loop dataset creation for a variety of nlp tasks. annotation was overseen by two experts in online hate. the annotation process is described in the following section. annotation guidelines were created at the start of the project and then updated after each round in response to the increased need for detail from annotators. we followed the guidance for protecting and monitoring annotator well-being provided by @xcite . 20 annotators were recruited. they received extensive training and feedback during the project. full details on the annotation team are given in appendix e. the small pool of annotators was driven by the logistical constraints of hiring and training them to the required standard and protecting their welfare given the sensitivity and complexity of the topic. nonetheless, it raises the potential for bias. we take steps to address this in our test set construction and provide an annotator id with each entry in our publicly-released dataset to enable further research into this issue. ## dataset formation the dataset was generated over four rounds, each of which involved ∼10, 000 entries. the final dataset comprises 41, 255 entries, as shown in target model implementation every round has a model in the loop, which we call the 'target model'. the target model is always trained on a combination of data collected in the previous round(s). for instance, m2 is the target model used in r2, and was trained on r1 and r0 data. for consistency, we use the same model architecture everywhere, specifically roberta @xcite with a sequence classification head. we use the implementation from the transformers @xcite library. more details are available in appendix d. for each new target model, we identify the best sampling ratio of previous rounds' data using the dev sets. m1 is trained on r0 data. m2 is trained on r0 data and r1 upsampled to a factor of five. m3 is trained on the data used for m2 and r2 data upsampled to a factor of one hundred. m4 is trained on the data used for m3 and one lot of the r3 data. ## model performance in this section, we examine the performance of models on the collected data, both when used inthe-loop during data collection (measured by the model error rate on new content shown by annotators), as well as when separately evaluated against the test sets in each round's data. we also examine how models generalize by evaluating them on the out-of-domain suite of diagnostic functional tests in hatecheck. ## discussion online hate detection is a complex and nuanced problem, and creating systems that are accurate, 4 see: https://www.perspectiveapi.com/#/ home . however, our approach also presents some challenges. first, it requires substantial infrastructure and resources. this project would not have been possible without the use of an online interface and a backend that can serve up state-of-the-art hate speech detection models with relatively low latency. second, it requires substantial domain expertise from dataset creators as well as annotators, such as knowing where to find real-world hate to inspire synthetic entries. this requires a cross-disciplinary team, combining social science with linguistics and machine learning expertise. third, evaluating and validating content in a time-constrained dynamic setting can introduce new pressures on the annotation process. the perturbation process also requires additional annotator training, or else might introduce other inadvertent biases. ## conclusion we presented a human-and-model-in-the-loop process for training an online hate detection system. it was employed dynamically to collect four rounds of hate speech datasets. the datasets are large and high quality, having been obtained using only expert annotators. they have fine-grained annotations for the type and target of hate, and include perturbations to increase the dataset difficulty. we demonstrated that the models trained on these dynamically generated datasets are much better at the task of hate speech detection, including evaluation on out-of-domain functional test suites. in future work we aim to expand the size and diversity of the annotator pool for further rounds of dynamic adversarial data collection. we would like to evaluate different models in-the-loop beyond roberta. the datasets also open many new avenues of investigation, including training models on only original entries and evaluating against perturbations (and vice versa) and training multi-label results for type and target of hate. data collection for future rounds is ongoing.