Automatically Exposing Problems with Neural Dialog Models (2021)

Abstract: Neural dialog models are known to suffer from problems such as generating unsafe and inconsistent responses. Even though these problems are crucial and prevalent, they are mostly manually identified by model designers through interactions. Recently, some research instructs crowdworkers to goad the bots into triggering such problems. However, humans leverage superficial clues such as hate speech, while leaving systematic problems undercover. In this paper, we propose two methods including reinforcement learning to automatically trigger a dialog model into generating problematic responses. We show the effect of our methods in exposing safety and contradiction issues with state-of-the-art dialog models.

## introduction language models, including dialog models, greatly benefit from training on large amounts of data with the objective of mimicking human generated sentences @xcite @xcite @xcite . however, even with carefully preprocessed training data from online sources, neural dialog models are prone to issues including generic utterances, repetition, contradiction, and lack of safety @xcite @xcite @xcite . compared to modularized dialog systems which are designed to avoid these problems @xcite , fixing these issues with end-to-end neural models is more challenging, which may hinder real world use of trained models @xcite . we argue that before solving these problems using simulated data from simplified scenarios, we need to be able to probe the models and expose the problems in a dynamic way. even though crucial limitations of neural dialog models are prevalent, they are mostly manually identified and categorized through interactions between model designers and the dialog system during qualitative analysis @xcite . recent work proposes asking annotators to converse with dialog models while goading the model into generating problematic responses in a black-box attack setting. although the data collected in this way can improve the performance of both problem classifiers and model generation, human annotators mostly rely on straightforward and intuitive strategies to collect the dataset, which may only expose superficial problems. for instance, @xcite instructs crowdworkers to trigger dialog systems into responding with unsafe (offensive or otherwise socially undesirable) utterances, but most of the human messages are either hate speech or controversial statements. similarly, @xcite asks mechanical turkers to manually write contradicting dialogs for both humans and bots, or to interact with chatbots, where a frequent strategy is to ask factual questions intentionally leading to contradiction (e.g. ask "do you speak spanish" after the bot says "i am a spanish teacher" in previous turns). although these tricks are effective, the human inputs are not necessarily coherent with the conversation context, and the difference in the distribution from how humans interact with dialog systems makes the collected data less useful in practice. in addition, the data collection procedure is extremely expensive and is not practical for newly trained models. more importantly, systematic problems are still not revealed. in this work, we propose to automatically expose problems with neural dialog models in a more systematic setting. given a conversation context, the goal is to generate a coherent utterance to act as a human prompt through self-chat, which will trigger the dialog system into generating a problematic response. to this end, we propose to learn some trigger hidden states while freezing the original dialog model. we assume that we have some problem classifiers which can be from in-domain x 1 y 1 x 2 y 2 ... x k-1 y k-1 x k ... y k-2 i prefer to watch it live, but if it's on tv, i only watch big games like the superbowl y k-1 i do, the dallas cowboys. do you have a favorite team? figure 1 : illustration of our problem exposure task and proposed model. given conversation history, our goal is to generate a coherent prompt x k , which will induce a neural dialog model (encoder-decoder in this example, all its parameters are frozen) to respond an utterance y k that contains problems such as unsafe and inconsistent. to do this, we learn hidden states h trigger which will guide the decoder to generate x k through attention. in the second step, we remove the learned hidden states and append newly generated utterance x k to generate a response y k . contextualized y k representation is sent to a problem classifier to output either gradients for trigger_weakly (which requires h trigger in step 2), or a reward for trigger_ppo. for trigger_ppo_adv, x k is also sent to the classifier to obtain a reward. or out-of-domain collected data. this is practical because out-of-domain data is relatively easy to collect and we do not require a perfect classifier. each token can attend to the trigger hidden states when generating the next tokens so that the generated prompt include systematic signals regarding target problems. specifically, we introduce a weakly-supervised method where we can back-propagate the gradients from the classifier through self-attention and cross-attention. we also introduce a reinforcement learning method that uses classifier results on the model responses as rewards 1 . compared to sentiment neurons @xcite , learning trigger hidden states as a problem switch is a much harder task because our hidden states in a relative shallow model are not trained with a huge amount of clearly distinguished supervised data. furthermore, exposing more subtle problems (such as contradiction) with a coherent prompt that will indirectly impact on the model response rather than direct conditional text generation is more challenging, similar to probing models in an adversarial attack setting. however, we demonstrate the effectiveness of our proposed methods on automatic problem exposure with the state-of-the-art chatbot blenderbot @xcite . we evaluate with two problems: safety and consistency. in addition, we show that the generated examples can help to improve the performance of out-of-domain problem classification as well. 1 our code is available at https://github.com/ diandyu/trigger 2 task definition given the context of a conversation c k-1 = x 1 , y 1 , x 2 , y 2 , ..., x k-1 , y k-1 , where x i , y i represents utterances from each speaker in a turn, we want to generate a prompt x k while keeping the whole conversation coherent and engaging. the original neural dialog model then considers the whole context (c k-1 ; x k ) to generate a response y k , which is considered as not acceptable to a problem p (e.g. toxic response to the safety problem). meanwhile, we have some trained classifier f p (h(y k )) for the problem p which can indicate how likely the contextual representation h(y k ) has the problem. ## methodology since the goal of the task is to expose systematic problems of pre-trained models rather than relying on simple tricks, we generate prompts using the same model in a self-chat paradigm so that when we plug in the generated prompts to the original model we get exactly the same response. compared to recent work on instructing humans to goad chatbots where annotators have no information of how the models works in a trial-and-error blackbox attack manner, we use gradients of the model. motivated by recent success in conditional generation without fine-tuning model parameters @xcite , we propose to learn a trigger prompt hidden states, h trigger , while freezing the original dialog model to maintain output distribution and generation quality. specifically, for an encoder-decoder model (or a language model), we are modeling in order to optimize h trigger which will boost the level of target problem through attention mechanism @xcite , we propose a weaklysupervised trigger model (section 3.1) where we backpropagate gradients from the classifier back to the hidden states directly, and a reinforcement learning trigger model (section 3.2) where the classifier results are used as rewards to optimize the hidden states. we illustrate the task and the proposed methods in figure 1 . ## experiments and results we evaluate our proposed approaches on two problems: safety and consistency by generating prompts that can trigger corresponding problems. in addition, we study whether our generated results can in turn improve the classification performance with out-of-domain data. for all our experiments, we use the state-of-theart open-domain chatbot blenderbot @xcite as our pre-trained neural dialog model. the maximum context and response lengths is set to 128 bpe tokens @xcite . blenderbot is pre-trained on reddit discussions @xcite ) with heuristic filtering and fine-tuned on human-collected clean conversational data including convai2 @xcite and blended skill talk @xcite . because of the finetuning data, the chatbot frequently deviates from the current conversation topic and asks simple questions such as "do you have a pet". this makes it even harder to generate unsafe and contradictory responses given a coherent prompt. for decoding, we follow the same procedure as in the original model, except that we use sampling instead of beam-search to increase diversity (which is shown to perform as well as beam search in their paper). during training, we set a maximum number of training steps with early stopping. to prevent unfair comparison to baselines, instead of selecting the best model based on average reward, we early stop when perplexity diverges too much from the original perplexity. please see appendix a.1 for implementation details. we analyze the effect of early stopping in section 5. ## related work for our introduced task to expose problems with pre-trained dialog models, the most relevant work is in the fields of controlled generation and adversarial attack. the goal for controlled generation is to generate coherent sentences containing some target attributes, whereas the task for adversarial attack is to craft some examples that can fool some trained classifiers. controlled generation most previous work in controlled text generation involve training or finetuning the whole model @xcite @xcite . alternatively, to utilize the high-quality pre-trained language model quality, @xcite ; @xcite propose to perturb token distributions towards a specific attribute with residual adapters @xcite . recently, @xcite ; @xcite show that optimizing simple prefix hidden states is effective in controlling pre-trained models, which inspires us to expose problems in neural dialog models by learning h_trigger. in terms of applying reinforcement learning to language generation tasks, previous work leverages defined reward functions @xcite @xcite or human preference @xcite . all these work targets at generating sentences that contain target attributes. in contrast, our work optimizes prompt generation, which indirectly triggers a pretrained model generating responses containing target attributes. there is no straightforward way to apply previous techniques to this task. adversarial attacks with pre-trained neural models similar to generating adversarial examples to fool natural language understanding models @xcite @xcite , @xcite ; @xcite show that some learned discrete nonsensical universal triggers can be used to generate unsafe sentences. on the other hand, gehman et al. (2020) finds toxic prompts from naturally occurring sentences. the most similar work to ours is probably directing pre-trained models into generating a list of pre-defined tokens or sentences @xcite @xcite . in comparison, our task needs to generate coherent prompts according to the conversation history. furthermore, instead of triggering pre-defined egregious responses, our proposed method is more flexible in exposing a wide range of problems such as consistency where crafting the target responses without context in advance is impossible. ## conclusion in this paper, we propose a weakly-supervised approach and a reinforcement learning approach to automatically expose problems with neural dialog models. compared to data annotated by humans that rely on simple tricks, our methods can expose more systematic problems with coherent prompts and can help find these problems easily with newly trained neural models. we conduct extensive experiments with a safety exposure task and a consistency exposure task, and show that our proposed methods are effective. in addition, we showed that our method can also be used to generate data to improve performance of out-of-domain problem classifiers. in the future, we plan to extend our methods to other problems such as generic utterances and hallucination @xcite .