C onv A buse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI (2021)

Abstract: We present the first English corpus study on abusive language towards three conversational AI systems gathered ‘in the wild’: an open-domain social bot, a rule-based chatbot, and a task-based system. To account for the complexity of the task, we take a more ‘nuanced’ approach where our ConvAI dataset reflects fine-grained notions of abuse, as well as views from multiple expert annotators. We find that the distribution of abuse is vastly different compared to other commonly used datasets, with more sexually tinted aggression towards the virtual persona of these systems. Finally, we report results from bench-marking existing models against this data. Unsurprisingly, we find that there is substantial room for improvement with F1 scores below 90%.

## introduction abusive language detection has received extensive attention for social media, (see e.g. @xcite , but far less within the context of conversational systems. as argued by unesco @xcite , detection and mitigation of abuse towards these (often anthropomorphised) ai systems is important in order to avoid reinforcement of negative gender stereotypes. following this report, several recent works have investigated possible abuse mitigation strategies (cercas @xcite @xcite . however, the results of these studies are non-conclusive as they are not performed with live systems nor with real users -mainly because of the absence of reliable abuse detection tools. the majority of currently deployed systems use simple keyword spotting techniques, (e.g. @xcite , which tend to produce a high number of false positives, such as cases in which the user expresses frustration, or use of profanities for emphasis, as well as false negatives, e.g. missing out on subtler forms of abuse @xcite . recently, @xcite ; @xcite released an abuse detection tool trained on wikipedia comments and crowd-sourced adversarial user prompts (the latter are not freely available). whereas in this work, ## related work most work on detecting harmful content such as offensiveness, toxicity, abuse, and hate speech (see @xcite for definitions), has focused on social media platforms, foremost twitter (e.g. @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite , or other social media platforms, including facebook @xcite , gab @xcite , and reddit @xcite . work has also been undertaken on data from comments on news media @xcite @xcite , chatrooms and discussion forums @xcite , wikipedia discussions @xcite @xcite @xcite , and message services such as whatsapp @xcite . meanwhile, there has been relatively little work on abuse detection for conversational ai. furthermore, much of the work that does exist in this area does not actually involve human-machine dialogue: @xcite ; @xcite use a classifier developed on wikipedia comments, which was further trained on adversarial prompts collected via crowd-sourcing. similarly, de los riscos and d'haro (2021) designed a chatbot to intervene against online hate speech, trained and evaluated on data from wikipedia and civil comments. those few studies that do report abuse detection results from genuine human-machine conversations tend not to include publicly released datasets. these include several submissions to the amazon alexa challenge foot_1 (cercas @xcite @xcite . as such, to the best of our knowledge, this is the first study to release a public dataset of human-machine conversations for the task in this domain. while we aim to detect abuse directed against any target, gender-based abuse has been identified as a particularly prevalent problem in conversational ai (cercas @xcite @xcite , and abuse detection systems have themselves been found to contain gender biases @xcite . misogyny and sexism detection has been applied to social media in binary @xcite and multi-class @xcite settings. we extend this to take an intersectional approach, analysing multiple types of abuse in a hierarchical mutli-label framework (see §3.3). ## the convabuse corpus we collected data from conversations between users and three different conversational ai sys-tems, which have different goals and properties. two of them are classed as chatbots, i.e. social, open-domain systems, while the other is a transactional, goal-oriented system. the first two systems listed below are text-based, whereas the last system is voice-based with a synthetic female-sounding voice. as such, two out of the three systems are female gendered, either by voice or name. ## discussion and conclusion in this work, we provide new insights regarding the detection and description of abusive language towards conversational agents, in terms of data, labelling and models. this may facilitate the release of large pre-trained conversational ai models that are safety-aware @xcite as well as potentially allow us to better detect abuse in human-human conversations. data. in compiling the convabuse corpus, we have compared differences between abusive phenomena in conversational ai and social media. in our domain, users appear to focus their abuse on the agents themselves rather than third parties or groups, with a far higher proportion of the abuse sexist and misogynystic in nature. annotations. unlike the majority of previous work, we use annotators who are members of the groups typically targeted by such abuse, and who have expertise in such issues. we also use a more fine-grained labelling scheme, which is able to capture the nuances of abuse, and is important for the downstream task of abusive language mitigation. we obtain similar results evaluating on these annotators separately and capturing individual view-points, even in a simple multi-class setting. in future work, we will experiment with modelling individual annotators in a multi-task framework. models and data. in our benchmarking experiments, we find that fine-tuning a bert model produces the highest f1 scores. however, in many settings, a simple linear classifier (svm) outperforms an mlp, supporting the findings of @xcite 's survey that svms tend to outperform neural methods on abusive language detection tasks. in this work, we present a small, focused dataset of high quality annotations, which are also informative for corpus study. we show that training on labelled in-domain data leads to better performance than similarly sized out-of-domain datasets, confirming the differences between the domains and highlighting the need for conversational data. while performance using general domain pretrained models leaves room for improvement, in future work, we hope to experiment with different initialisation settings, using models trained on data and tasks more similar to those of convabuse, such as hatebert @xcite or hurtbert @xcite .