Robust Conversational Agents against Imperceptible Toxicity Triggers (2022)

Abstract: Warning: this paper contains content that maybe offensive or upsetting. Recent research in Natural Language Processing (NLP) has advanced the development of various toxicity detection models with the intention of identifying and mitigating toxic language from existing systems. Despite the abundance of research in this area, less attention has been given to adversarial attacks that force the system to generate toxic language and the defense against them. Existing work to generate such attacks is either based on human-generated attacks which is costly and not scalable or, in case of automatic attacks, the attack vector does not conform to human-like language, which can be detected using a language model loss. In this work, we propose attacks against conversational agents that are imperceptible, i.e., they fit the conversation in terms of coherency, relevancy, and fluency, while they are effective and scalable, i.e., they can automatically trigger the system into generating toxic language. We then propose a defense mechanism against such attacks which not only mitigates the attack but also attempts to maintain the conversational flow. Through automatic and human evaluations, we show that our defense is effective at avoiding toxic language generation even against imperceptible toxicity triggers while the generated language fits the conversation in terms of coherency and relevancy. Lastly, we establish the generalizability of such a defense mechanism on language generation models beyond conversational agents.

## introduction adversarial attacks on different machine learning (ml) and natural language processing (nlp) applications can reveal important vulnerability issues related to these systems. most existing research focuses on adversarial attacks that degrade performance of existing ml systems with regards * currently at google research. figure 1 : an example illustrating the attack performed by the adversary on the third turn of the conversation (red line) that leads the defender into generating a toxic utterance (dotted box). with a proper defense the defender can bypass the attack and generate a non-toxic response (green line). to accuracy @xcite . more recent work has considered attacks that target ethical concerns, such as triggering the models into outputting unfair predictions @xcite , or in the context of nlp, generating biased @xcite and toxic @xcite text. in this paper, we consider adversarial attacks on human-centric chatbots and dialogue systems. it is important for these systems to be safe and robust in the face of natural(-looking) human conversations. further, the defender should ensure a satisfying user experience via relevant and coherent generation. an instance of the attack and defense is demonstrated in figure 1 in which the adversary tries to trigger the defender while the defender avoids the attack by not generating toxic utterances. 1 the existing work on adversarial attacks on language generation is relatively thin. @xcite offer attacks based on universal adversarial triggers (uat) that can result in toxic text generation with a relatively high success rate. how-ever, those triggers are unnatural, incoherent sequences of words that can be easily detected via a language model loss. furthermore, such attacks cannot be successful in voice-based dialogue systems where the input to the dialogue model comes from speech recognition and should necessarily conform to human language norms. @xcite use human-and-model-in-the-loop framework to generate natural-looking attacks to break chatbots, but this approach is costly and inherently not scalable. in this paper, we propose imperceptible adversarial attacks on dialogue systems that leverage natural-looking and coherent utterances as triggers, which cannot be easily detected using anomaly detection techniques. as such, these attacks can also target voice-based assistants who see the world through the lens of speech recognition systems. our proposed approach works by augmenting the uat from @xcite with additional selection criteria to generate imperceptible yet effective triggers. the method is fully automated and scalable, thus affording the exploration of a large number of attack vectors and system vulnerabilities efficiently. through human and automatic evaluations we show the effectiveness of the proposed attack in provoking the defender into generating toxic responses while keeping the fluency and coherency of the conversation intact. we then focus on a defense mechanism for the non-adversarial (defender) model to avoid generating toxic utterances. while simple defense methods such as @xcite achieve near-perfect effectiveness against adversarial triggers, those methods work by essentially resetting the conversation topic which breaks the flow. instead, we are interested in a defense mechanism that "detoxifies" the response while preserving the natural conversation flow. our proposed method relies on two levels of interpretable reasoning that helps the model to ( 1 ) identify the key adversarial tokens responsible for the attack and (2) avoid generating toxic responses by masking those tokens during the generation process. we perform automatic and human evaluations to assess the effectiveness of our defense mechanism and demonstrate that it compares favorably with various state of the art baselines, both in terms of detecting the attacks and generating conversationally fluent responses. we finally demonstrate the generalizability of such a defense mechanism on generation tasks beyond conversational models. we emphasize that while our problem formulation focuses on the adversarial scenario, the imperceptible and coherent-looking triggers used in our proposed attacks can also be invoked inadvertently by regular (non-adversarial) users. thus, the defense mechanism proposed against such triggers will improve the overall robustness of conversational agents, not only against adversaries but also in interactions with regular users. ## attack approaches in this section, we first discuss the universal adversarial trigger attack proposed by @xcite , which we use as our baseline. we then propose alterations to this baseline to make the universal triggers more natural-looking and suitable for conversational domain. finally, we discuss our performed experiments and results. ## defense approaches the defense against adversarial attacks has two components (a) detecting the attack and (b) mitigating its effect by ensuring that the defender does not generate a toxic response. the detection problem is rather straightforward, as the defense can simply run a toxicity classifier on the generated response. the mitigation is more challenging. @xcite suggested a mitigating approach which, when a toxic response is detected, simply resets the dialogue and generates a (non-toxic) utterance by randomly sampling from a predefined set of topics (see section 3.2.1). as we mentioned before, we are interested in mitigation strategies that avoid generating toxic utterances but at the same time manage to keep the conversation flow intact. we now discuss our approach in more details. ## beyond conversational agents we show the generalizability of our defense method against non-conversational generation tasks, by conducting experiments with realtoxicityprompts dataset @xcite . previous work showed that the prompts in realtoxicityprompts can force different generative models such as gpt-2 @xcite to generate toxic responses. thus, we used our defense to test whether it can also be effective in reducing the number of toxic responses given these prompts in realtoxic-ityprompts in the gpt-2 model. as evident from the previous discussions, the non sequitur baseline defense @xcite that we considered in our paper, only works for the conversational domain; however, our method has the advantage of working on any conditional generation task. we used the 100k prompts in realtoxicityprompts and reported the number of toxic generations before and after applying our defense from the gpt-2 model. results in figure 9 demonstrate that one iteration of our defense reduces the number of generated toxic responses by 81%, 31%, and 23%, according to toxic-bert, perspective api, and safety classifier, respectively. although the defense is based on figure 9 : number of generated toxic responses before and after the defense was applied to gpt-2 from the re-altoxicityprompts dataset @xcite . our defense is shown to reduce the number of toxic generations in gpt-2. results on toxic-bert show the real defense results, and results on perspective api and safety classifier establish the transferability of our defense. toxic-bert, the results still transfer to perspective api and safety classifier. these results show the effectiveness of our defense in reducing toxic generations beyond conversational domain and a step toward reducing toxic generation. notice that the setup of this experiment was not adversarial; however, prompts were causing the toxic generations. ## related work crafting adversarial examples and using them in training was previously shown to be an effective technique in improving nlp and ml models @xcite @xcite . not only that, but adversarial attacks can reveal important vulnerabilities in our systems @xcite . although previous work has studied adversarial examples in nlp @xcite @xcite most of them focused on accuracy as a metric of interest. among the ones that studied toxicity and other ethical considerations @xcite they did not put the focus on either conversational agents or they did not consider attacks being imperceptible. @xcite ; @xcite studied adversarial attacks on conversational agents; however, their focus was on task oriented dialogue systems and also did not consider toxicity but accuracy as a metric. @xcite also considered conversational domains; however, they relied on human adversaries which can be costly and non-scalable. beyond attacks, we discussed a possible defense mechanism to improve robustness of generative models against generating toxic responses using interpretability methods. using interpretability mechanisms was also previously shown to be effective in reducing bias in ml applications @xcite . in addition, there is a body of work in detecting toxic behavior in conversational agents @xcite @xcite that can be utilized to design ethically aligned systems. ## conclusion we studied the possibility of generating imperceptible attacks against conversational agents that, while fluent and coherent, target the model into generating toxic responses. through various automatic and human experiments, we showed the effectiveness of our attacks both in terms of being adversarial as well as being able to maintain coherency, relevancy, and fluency of the generated conversation (what we referred as the imperceptibility of the attack). we then proposed a defense mechanism that was shown to be effective through various automatic and human evaluations as well as its transferability to human attacks, general generation tasks, and different toxicity classifiers. future work can focus on improving our proposed attacks both in terms of imperceptibility and effectiveness as well as more advanced defense mechanisms.