Large-Scale Quantitative Evaluation of Dialogue Agents’ Response Strategies against Offensive Users (2021)

Abstract: As voice assistants and dialogue agents grow in popularity, so does the abuse they receive. We conducted a large-scale quantitative evaluation of the effectiveness of 4 response types (avoidance, why, empathetic, and counter), and 2 additional factors (using a redirect or a voluntarily provided name) that have not been tested by prior work. We measured their direct effectiveness on real users in-the-wild by the re-offense ratio, length of conversation after the initial response, and number of turns until the next re-offense. Our experiments confirm prior lab studies in showing that empathetic responses perform better than generic avoidance responses as well as counter responses. We show that dialogue agents should almost always guide offensive users to a new topic through the use of redirects and use the user’s name if provided. As compared to a baseline avoidance strategy employed by commercial agents, our best strategy is able to reduce the re-offense ratio from 92% to 43%.

## introduction conversational bots are increasingly popular among the general population which is correlated with an increase in bot abuse (cercas @xcite . analysis of the chat logs of an alexa prize foot_0 competition social bot shows that more than 10% of the conversations contain some level of offensiveness. recently, researchers begin to measure the appropriateness of virtual agent responses to abuse. however, prior work either use selfreported scales of emotions by non-anonymous volunteers @xcite or perceived quality of the conversation from crowd workers (cer-cas @xcite . however, these qualitative metrics only measure the appropriateness of the response rather than the actual effect of the responses in a real conversation. unlike the participants recruited for controlled lab studies or crowdsourced studies, real users abuse agents voluntarily, anonymously, and repeatedly. to address these limitations, we conducted a large scale study similar to @xcite to quantitatively measure the effectiveness of response strategies. as opposed to @xcite which uses user ratings as the evaluation metric, we measured 1) the re-offense ratio; 2) the number of turns until the next offense; 3) the number of turns until the end of the conversation after the initial response. these metrics measure offensive behavior directly as opposed to user ratings which measures the quality of conversations as a whole. we show that using a redirection is significantly better than not using one, and using empathetic responses and user names is also effective at mitigating abuse, but only in combination with a redirection. ## related work there's a large body of research on physical agent abuse @xcite , particularly by children @xcite @xcite @xcite . there has also been much work on understanding the reason behind bot abuse @xcite @xcite . more recently, cercas @xcite found that "polite refusal" responses are the most appropriate compared to many other responses by commercial bots. similarly, @xcite ; @xcite further evaluated the effectiveness of empathetic and counter-attacking response strategies by measuring their impact on cultivating emotions that are known to reduce ag- ## hypotheses we test 4 hypotheses in our work: in order to test these hypotheses as well as interactions between different factors influencing the effectiveness of the response strategies, we cross multiple conditions with each other. full description can be found at ## data collection we built our experiments into a custom opendomain conversational chatbot developed as part of the alexa prize competition. during the competition, alexa users can invoke a competition bot by saying "alexa lets chat" or just "lets chat" to an alexa-enabled device, after which alexa hands off the conversation to a randomly assigned competition bot. ## proposed metrics we propose 3 metrics that directly measure strategy effectiveness from conversation logs. the first metric is the re-offense ratio (a.k.a. re-offense), measured as the number of conversations that contained another offensive utterance after the initial bot response over the total number of conversations that used the same strategy. intuitively, the responses leading to a smaller number of re-offenses more are effective at reducing user abuse. we also measure the length of the conversation after the response assuming there are no more re-offenses (a.k.a. end) to understand how a strategy stopped abuse. when the strategy is unable to stop re-offense, we are interested to know how many turns passed until the user offended again (a.k.a. next). we believe that strategies that are able to delay offense longer are more effective at mitigating user abuse. ## hypothesis testing and discussion all the metrics measured are shown in ## future directions the main limitation of our work was keeping customer satisfaction in mind when designing our experiments under alexa prize competition rules. this prevented us from replicating strategies such as joke strategies mentioned in cercas @xcite and parenting strategies such as love-withdrawal as mentioned in @xcite . we were similarly unable to test the effectiveness of de-anonymization and peer-listening strategies similar to tan et al. (2018) that would test how would the users respond if they were told that their conversations were not anonymous/private. it would also be useful to gather metadata about our participants such as age and gender (while maintaining anonymity). however, this is not allowed under alexa prize competition rules. ## ethical concerns despite the empirical effectiveness of the avoidance + redirect strategy as detailed in this work, we would like to remind researchers of the societal dangers of adopting similar strategies. alexa has a default female voice and the majority of offensive responses we receive are sexual in nature as stated before. as pointed out by prior work @xcite @xcite , inappropriate responses further gender stereotypes and set unreasonable expectations of how women would react to verbal abuse. without pointing out the inappropriateness of user offenses, these response strategies could cause users to believe their offenses will go unnoticed in the real world as well. thus, we urge researchers to consider the greater impact of deploying such strategies in voice-based dialogue agents beyond the proposed effectiveness metrics. ## conclusion we present the first study on automatically measuring conversational agent offense mitigation strategies in-the-wild using 3 intuitive and novel metrics: re-offense ratio, length of the conversation after bot response, and number of turns until the next offensive utterance. we believe the automatic metrics we proposed make it easier to quickly evaluate response strategies, and thus allow researchers to experiment with more factors for constructing a successful response. we evaluated 4 response strategies (avoidance, why, empathetic, and counter) with 2 additional factors (redirect and name). we showed that to mitigate offensiveness, the bot should almost always empathetically and actively move on to a different topic, and while doing so use the offending user's name whenever possible. we found that the bot should never ask a user why they made offensive utterances, as doing so causes the user to almost always repeat their offense immediately. we hope our systematic evaluation of response strategies raises awareness of bot abuse as social bots become more popular and accessible.