P rompt A ttack: Probing Dialogue State Trackers with Adversarial Prompts (2023)

Abstract: A key component of modern conversational systems is the Dialogue State Tracker (or DST), which models a user’s goals and needs. Toward building more robust and reliable DSTs, we introduce a prompt-based learning approach to automatically generate effective adversarial examples to probe DST models. Two key characteristics of this approach are: (i) it only needs the output of the DST with no need for model parameters, and (ii) it can learn to generate natural language utterances that can target any DST. Through experiments over state-of-the-art DSTs, the proposed framework leads to the greatest reduction in accuracy and the best attack success rate while maintaining good fluency and a low perturbation ratio. We also show how much the generated adversarial examples can bolster a DST through adversarial training. These results indicate the strength of prompt-based attacks on DSTs and leave open avenues for continued refinement.

## introduction task-oriented dialogue systems aim to help users with tasks through a natural language conversation. example tasks include booking a hotel or completing a do-it-yourself project. a key component for enabling a high-quality task-oriented dialogue system is the dialogue state tracker (or dst) which plays an important role in understanding users' goals and needs @xcite @xcite @xcite @xcite . for example in figure 1a , given the user utterance "i am looking for a cheap restaurant in the center of the city", the dst extracts the user's preference for booking a restaurant, which is typically represented as slot-value pairs such as (restaurant-price range, cheap) and @xcite . the current state of the conversation is a primary driver of the subsequent dialogue components (e.g., what is the next action to take? what is the appropriate response to generate?). ## related work adversarial examples have been widely explored to investigate the robustness of models @xcite . recent work in the nlp domain has targeted tasks like text classification and inference @xcite @xcite @xcite @xcite , reading comprehension @xcite , named entity recognition @xcite , and machine translation @xcite . these works typically aim to construct examples that are imperceptible to human judges while misleading the underlying model to make an incorrect prediction, while also maintaining good fluency and semantic consistency with original inputs @xcite . only a few works have begun to explore adversarial examples in dsts like coco @xcite , which aims to test the robustness of models by creating novel and realistic conversation scenarios. they show that dst models are susceptible to both unseen slot values generated from in and out of the slot domain. @xcite propose a model-agnostic toolkit to test the robustness of task-oriented dialogue systems in terms of three aspects: speech characteristics, language variety, and noise perturbation. the adversarial examples are based on heuristics and it is unclear how to adapt such an approach to new victim models effectively without more hand-crafted templates. in contrast, we explore in this paper the potential of a learning-based approach to generate effective adversarial examples. prompt learning is a recently proposed paradigm for using prompts to better probe and adapt large pre-trained language models (plms) to a variety of nlp tasks, e.g., text classification and inference @xcite @xcite , factual probing @xcite , summarization @xcite , and dialogue systems @xcite @xcite . with the increase in the size of plms, prompt learning has been shown to be parameter-efficient @xcite @xcite . there are two types of prompts: discrete (or hard) prompts and continu- change fill in belief states: [s] = [v] belief states: restaurant-price @xmath0. restaurant-price range, cheap restaurant-price range, expensive template discrete prompt belief states (a) discrete prompt construction. the discrete prompt is constructed by filling pre-designed templates with slots extracted from the dst model and corresponding random values. dst model loss prompt dialogue (b) continuous prompt tuning. the continuous prompt is prepended before the dialogue context embeddings and tuned by optimizing the loss while keeping dst model parameters fixed. prompt dialogue black-box dst model adversarial utterances mlm model mask (c) adversarial example generation. the adversarial prompt (discrete or continuous) is prepended before the masked dialogue context (or embeddings) to generate perturbations via mask-and-filling. after removing the adversarial prompt, the generated adversarial example is used to attack victim models. ous (or soft) prompts. discrete prompts are humandesigned text strings @xcite while continuous prompts are continuous embeddings. soft prompts proposed by @xcite prepend a sequence of continuous vectors to the input, freeze the language model parameters, and then backpropagate the error during tuning. in this paper, we explore both approaches in the design of our prompt-based attack framework. recent works have begun to explore how prompts can be helpful in exposing fundamental flaws in large language models. @xcite shows how to manually design prompts to flip the output of a model for classification tasks. however, it is time-consuming to design and find prompts that are most effective to generate adversarial examples capable of successfully attacking victim models. it is an open question how to leverage prompts for uncovering effective adversarial prompts. ## promptattack our prompt-based learning approach proceeds in two stages. first, our goal is to identify adversarial prompts that can effectively probe a dst to reveal gaps in its robustness. in the second, we use these prompts to create adversarial examples that can attack dsts successfully while maintaining good fluency. figure 2 shows an overview of the proposed approach. in the following, we first formalize dsts and the problem of probing a dst. then, we introduce the details of promptattack. attacking a dst. given dialogue history c t , current user utterance u t , and dialogue belief states b t , the purpose of an adversarial attack on a dst is to intentionally perturb the original user utterance u t to get an adversarial example u ## experimental setup our experiments are designed to test the effectiveness of the proposed prompt-based approach to attack dst models. we structure the experiments around four research questions: rq1: are adversarial examples learned by promptattack effective and transferable? and how do these examples compare against baseline (non-prompt) approaches? rq2: are the generated adversarial examples of good quality? that is, are they fluent with a low perturbation ratio? rq3: what impact do the design choices of promptattack have, i.e., the ratio of perturbed tokens and prompt length? rq4: and finally, can the generated adversarial examples be used to improve the performance of current dst models to improve their robustness? ## experimental results given this setup, we now investigate the four experimental research questions in turn. ## conclusion in this paper, we present a prompt-based learning approach that can generate effective adversarial examples for probing dst models. through experiments over four state-of-the-art dsts, our framework achieves the greatest reduction in accuracy with the best attack success rate. moreover, the generated adversarial examples maintain good fluency and low perturbation ratio, evidence that they are close to legitimate non-adversarial user inputs. we also show our generated adversarial examples can bolster a dst by augmenting the original training data with adversarial examples. we find that both discrete and continuous adversarial prompts are capable of generating effective adversarial examples. discrete prompts are more interpretable while continuous prompting allows us to search for optimal adversarial prompts more efficiently, and generates more effective adversarial examples.