A Generative User Simulator with GPT -based Architecture and Goal State Tracking for Reinforced Multi-Domain Dialog Systems (2022)

Abstract: Building user simulators (USs) for reinforcement learning (RL) of task-oriented dialog systems (DSs) has gained more and more attention, which, however, still faces several fundamental challenges. First, it is unclear whether we can leverage pretrained language models to design, for example, GPT-2 based USs, to catch up and interact with the recently advanced GPT- 2 based DSs. Second, an important ingredient in a US is that the user goal can be effectively incorporated and tracked; but how to flexibly integrate goal state tracking and develop an end-to-end trainable US for multi-domains has remained to be a challenge. In this work, we propose a generative user simulator (GUS) with GPT-2 based architecture and goal state tracking towards addressing the above two challenges. Extensive experiments are conducted on MultiWOZ2.1. Different DSs are trained via RL with GUS, the classic agenda-based user simulator (ABUS) and other ablation simulators respectively, and are compared for crossmodel evaluation, corpus-based evaluation and human evaluation. The GUS achieves superior results in all three evaluation tasks.

## introduction task-oriented dialog (tod) systems are mainly designed to help users accomplish specific goals, such as finding restaurants or booking flights. the dialog system (ds) usually consists of several modules -dialog state tracking (dst), database querying (db), dialog policy (dp) and natural language generation @xcite . recent studies recast these modules all as conditional generation of tokens and build on some pretrained language model (plm) such as gpt-2 @xcite as the backbone. fine-tuning plm over annotated dialog datasets via supervised learning (sl) has shown state-ofthe-art results @xcite , 2020; @xcite @xcite , thanks to the powerful generation ability of plms. however, supervised trained agents could become biased by the annotations, and it has long been recognized that reinforcement learning (rl) could be applied to policy learning for the agent @xcite , which aims at goal-directed learning from interaction between the dialog agent and the user. interaction with human users is expensive and time-consuming in practice. therefore, an alternative approach, building user simulators (uss), has gained more and more attention, which, however, still faces several fundamental challenges. first, note that the recent research on building dialog agents has been significantly advanced with the end-to-end trainable generative approach based on plms such as gpt-2. however, prior work on user simulators are mostly lstm-based, not utilizing any plms, as reviewed in second, an important ingredient in a us is that the user goal can be incorporated and tracked. taskoriented dialog systems are characterized by a user goal, which is composed of user constraints and requests. the user goal ensures that the user behaves in a consistent, goal-directed manner, and the system agent is considered successful if it is able to fulfill the user goal by the end of a dialog session. thus, it is desirable for the us to track the completion process of the goal explicitly (which we call goal state tracking in this paper), as did in the classic agenda-based user simulator (abus) @xcite . however, the goal state tracking process is overlooked in later data-driven uss @xcite @xcite , or realized by binary vectors @xcite @xcite , or only works at the semantic level @xcite . how to flexibly integrate goal state tracking and develop an end-to-end trainable us for multi-domains has remained to be a challenge. in this work, we propose a generative user simulator (gus) with gpt-2 based architecture and goal state tracking towards addressing the above two challenges in building end-to-end trainable uss for reinforced multi-domain dialog systems. basically, a us, interacting with a ds in natural languages, needs several modules -natural language understanding (nlu) of system responses, goal state tracking (gst) to refresh the remained constrains and requests that need to send subsequently, user policy (up), and natural language generation (nlg). the information flow in a task-oriented dialog between a us and a ds is illustrated in figure 1 . in generative user simulator (gus), we recast these modules in us all as conditional generation of tokens, similar to the recent approach of finetuning plms such as gpt-2 to build end-to-end trainable generative dss. to be specific in this paper, we use the gpt-2 based architecture for gus to generate user acts and user utterances, and constantly track the goal 1 it can be seen that the discussion and the proposed method in the remainder of this paper can also be applied to other plms such as t5 @xcite , not limited to gpt-2. us plm goal state cross-model compared natural lang. multi-tracking evaluation with ds-sl interaction domain schatzmann et al. (2007) n y n n n n asri et al. (2016) n n n n n n liu and lane (2017) n n n y y n gür et al. (2018) n n n n n n kreyssig et al. (2018) n y y n y n papangelis et al. (2019) n n n y y n shi et al. (2019) n n y n y n takanobu et al. (2020) n y n y n y lin et al. (2021) n y y n n y tseng et al. (2021) n y n y y y gus y y y y y y in this work, the definition of goal state is similar to the agenda in abus @xcite , which represents a collection of pending user acts that are needed to elicit the information specified in the goal. the maintenance of goal state includes not only removing the completed user acts, but also changing the user goal when the system cannot find a requested entity. extensive experiments are conducted on mul-tiwoz2.1 @xcite . different dss are trained via rl with gus, abus and other ablation simulators respectively, and are compared for cross-model evaluation, corpus-based evaluation and human evaluation. the gus achieves superior results in all three evaluation tasks. us architecture a variety of user simulators have been studied, either rule-based or data-driven. a classic rule-based us is the agenda-based user simulator (abus) @xcite . different data-driven us models are proposed with different architectures and characteristics. asri et al. (2016) 2019 ) make extensive comparisons for six user simulators, based on two user policy modules (seq2seq or agenda based) and three nlg modules (template, retrieval or seq2seq). tus in (lin et al., 2021) designs domain-independent features and implements the user policy as multi-class classification so that tus could be easily adapted to new domains. some studies aim to jointly optimize ds and us. the uss used in these studies are mostly based on lstm seq2seq architectures @xcite @xcite , or simply as multi-class classification for action selection with feed-forward networks @xcite . goal state tracking in us abus is classic in goal state tracking, where the pending user acts are tracked in a stack-like structure, called agenda. abus is rule-based, generating user acts by pushing and popping hand-crafted rules from agenda. the goal state tracking process is overlooked in some later studies of data-driven uss @xcite @xcite , where the us is always conditioned on the whole initial user goal at each turn. some data-driven uss explicitly track goal states but employ binary vectors @xcite @xcite . the us in @xcite represents goal states by tokens, which is flexible, but the us only interacts with the ds at the semantic level (not end-to-end trainable). ## preliminaries notations according to the information flow in a task-oriented dialog between a us and a ds as illustrated in figure 1 , we let g t denote the user goal state, a u t the user act, u t the user utterance, b s t the system belief state, db t the database result, a s t the system act, and r t the system response, respectively, at turn @xmath0, • • • , t , for a dialog of t turns. moreover, in this paper we are interested in building end-to-end trainable us that can interact with the ds in natural languages. thus, we introduce a nlu module in the us, which takes the system response r t as input and infer system intent. the nlu result is denoted by b u t , or loosely speaking, referred to as the user belief state. notably, the us belief state b u t denotes the us's understanding only about the previous system response, and accordingly is labeled as a s t-1 in training. b u t is not of accumulated nature, since the us uses the goal state g t to summarize the dialog history encountered by the us foot_1 . ## conclusion in this paper, towards developing an end-to-end trainable us for multi-domains, a generative user simulator (gus) with gpt-2 based architecture and goal state tracking is proposed and systematically evaluated. we train gpt-2 based dss and uss and conduct cross-model evaluation, corpusbased evaluation and human evaluation. the results show that the ds trained with gus outperforms both the supervised trained ds and the ds trained with abus. the human evaluation further confirms the superiority of gus and shows that gus can generate much more coherent and fluent utterances than abus. moreover, gus is simple and easy to use, in addition to its strong performance. hope this work will stimulate further work on developing and using user simulators in the study of building dialog systems.