We present a first-of-its-kind large synthetic training dataset for online hate classification,created from scratch with trained annotators over multiple rounds of dynamic data collection. We provide a 40,623 example dataset with annotations for fine-grained labels , including a large number of challenging contrastive perturbation examples. Unusually for an abusive content dataset, it comprises 54\% hateful and 46\% not hateful entries . We show that model performance and robustness can be greatly improved using the dynamic data collection paradigm. The model error rate decreased across rounds , from 72.1\% in the first round to 35.8\% in the last round, showing that models became increasingly harder to trick -- even though content become progressively more adversarial as annotators became more experienced. Hate speech detection is an important and subtle problem that is still very challenging for existing AI methods. We hope that the models , dataset and dynamic system that we present here will help improve current approaches, having a positive social impact.
We present a human-and-model-in-the-loop process for dynamically generating datasets and training better performing and more robust hate detection models. We provide a new dataset of ~40,000 entries, generated and labelled by trained annotators over four rounds of dynamic data creation. It includes ~15,000 challenging perturbations and each hateful entry has fine-grained labels for the type and target of hate. Hateful entries make up 54\% of the dataset, which is substantially higher than comparable datasets . We show that model performance is substantially improved using this approach. Models trained on later rounds of data collection perform better on test sets and are harder for annotators to trick. They also perform better on HateCheck, a suite of functional tests for online hate detection. We provide the code , dataset and annotation guidelines for other researchers to use. Accepted at ACL 2021.
R
first-of-its-kind large synthetic training dataset for online hate classification,created from scratch with
human-and-model-in-the-loop process for dynamically generating datasets and training better performing and more robust hate detection models. We provide a new dataset of ~40,000 entries, generated and labelled by
R
multiple
four
R
collection. We provide a 40,623 example dataset with annotations for
creation. It includes ~15,000 challenging perturbations and each hateful entry has
R
, including a large number of challenging contrastive perturbation examples. Unusually for an abusive content dataset, it comprises
for the type and target of hate. Hateful entries make up
R
hateful and 46\% not hateful entries
of the dataset, which is substantially higher than comparable datasets
R
and robustness can be greatly improved using the dynamic data collection paradigm. The model error rate decreased across rounds , from 72.1\% in the first round to 35.8\% in the last round, showing that models became increasingly harder to trick -- even though content become progressively more adversarial as annotators became more experienced. Hate speech detection is an important and subtle problem that is still very challenging for existing AI methods. We hope that the models
is substantially improved using this approach. Models trained on later rounds of data collection perform better on test sets and are harder for annotators to trick. They also perform better on HateCheck, a suite of functional tests for online hate detection. We provide the code
R
dynamic system that we present here will help improve current approaches, having a positive social impact.
annotation guidelines for other researchers to use. Accepted at ACL 2021.