Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems (2020)

Abstract: Dialogue policy learning for task-oriented dialogue systems has enjoyed great progress recently mostly through employing reinforcement learning methods. However, these approaches have become very sophisticated. It is time to re-evaluate it. Are we really making progress developing dialogue agents only based on reinforcement learning? We demonstrate how (1) traditional supervised learning together with (2) a simulator-free adversarial learning method can be used to achieve performance comparable to state-of-the-art reinforcement learning-based methods. First, we introduce a simple dialogue action decoder to predict the appropriate actions. Then, the traditional multi-label classification solution for dialogue policy learning is extended by adding dense layers to improve the dialogue agent performance. Finally, we employ the Gumbel-Softmax estimator to alternatively train the dialogue agent and the dialogue reward model without using reinforcement learning. Based on our extensive experimentation, we can conclude the proposed methods can achieve more stable and higher performance with fewer efforts, such as the domain knowledge required to design a user simulator and the intractable parameter tuning in reinforcement learning. Our main goal is not to beat RL with supervised learning, but to demonstrate the value of rethinking the role of reinforcement learning and supervised learning in optimizing task-oriented dialogue systems.

## introduction the aim of dialogue policies in task-oriented dialogue system (tds) is to select appropriate actions at each time step according to the current context of the conversation and user feedback @xcite . in early work, dialogue policies were manually designed as a set of rules that map the dialogue context to a corresponding system action @xcite . the ability of rule-based solutions is limited by the domain complexity and task scalabil-ity. moreover, the design and maintenance of these rules require a lot of effort and domain knowledge. due to recent advantages in deep learning and the availability of labeled conversational datasets, supervised learning can be employed for dialogue policy training to overcome the disadvantages of rule-based systems. the downside of the supervised learning approach is that the dialogues observed in the datasets are unlikely to represent all possible conversation scenarios; in some extreme cases, the required conversational dataset cannot be collected or acquiring it might cost-prohibitive. the success of rl in other areas holds promises for dialogue policy learning (pl) @xcite . using rl techniques, we can train dialogue policies and optimize automatically, from scratch and utilizing interactions with users @xcite . in rl-based solutions, the dialogue system takes actions that are controlled by the dialogue policy, and user feedback (the reward signal), which is provided when the dialogue is finished, is utilized to adjust the initial policy @xcite @xcite . in practice, reward signals are not always available and may be inconsistent @xcite . as it is not practical to ask for explicit user feedback for each dialogue during policy training, different strategies have been proposed to design a rule-based user simulator along with a reward function that can approximate the real reward function which exists only in each user's mind. designing an appropriate user simulator and accurate reward function requires strong domain knowledge. this process has the same disadvantages as rule-based dialog systems @xcite . the difference is that rule-based approaches to system design meet this problem at the dialogue agent side while rule-based user simulators need to solve it at the environment side. if the task is simple and easy to solve, why not just build a rule-based system rather than a usersimulator that is then used with rl techniques to train the dialogue system, where more uncontrollable factors are involved? and if the task domain is complex and hard to solve, is it easier to design and maintain a complicated rule-based user simulator than to build a rule-based dialogue agent? supervised learning methods do not suffer from these issues but require labeled conversational data; in some exceptional cases, if the data cannot be collected for privacy reasons, rl is the solution. however, collecting labeled data is feasible for many applications @xcite @xcite . therefore in this work seek to answer the following research question: are we really making progress in tdss focusing purely on advancing rl-based methods? to address this question, we introduce three dialogue pl methods which do not require a user simulator. the proposed methods can achieve comparable or even higher performance compared to sota rl methods. the first method utilizes an action decoder to predict dialogue combinations. the second method regards the dialogue pl task as a multi-label classification problem. unlike previous work, we assign a dense layer to each action label in the action space. based on the second method, we propose an adversarial learning method for dialogue pl without utilizing rl. to backpropagate the loss from the reward model to the policy model, we utilize the gumbel-softmax to connect the policy model and the reward model in our third method. we compare our methods with rl and adversarial rl based dialogue training solutions to show how we can achieve comparable performance without a utilizing costly user simulator. to summarize, our contributions are: • a dialogue action decoder to learn the dialogue policy with supervised learning. ## related work a number of rl methods, including qlearning @xcite @xcite @xcite and policy gradient methods @xcite , have been applied to optimize dialogue policies by interacting with real users or user simulators. rl methods help the dialogue agent is able to explore contexts that may not exist in previously observed data. a key component in rl is the quality of the reward signal used to update the agent policy. most existing rl-based methods require access to a reward signal based on user feedback or a pre-defined one if feedback loop is not possible. besides, designing a good reward function and a realistic user simulator is not easy as it typically requires strong domain knowledge, which is similar to the problem that rule-base methods meet. peng et al. (2018a) propose to utilize adversarial loss as an extra critic in addition to the main reward function based on task completion. inspired by the success of adversarial training in other nlp tasks, liu and lane (2018) propose to learn dialogue rewards directly from dialogue samples, where a dialogue agent and a dialogue discriminator are trained jointly. following the success of inverse reinforcement learning (irl) in different domains, takanobu et al. (2019) employ adversarial irl to train the dialogue agent. they replace the discriminator in gail (ho and ermon, 2016) with a reward function with a specific architecture. the learned reward function can provide a stable reward signal and adversarial training can benefit from high quality feedback. compared to existing rl based methods, we propose strategy that can eliminate designing a user simulator and sensitive parameter-tuning process while bringing a significant performance improvement with respect to a number of metrics. the absence of user simulators involved will largely reduce the required domain knowledge and supervised learning can lead to robust agent performance. 3 multi-domain dialogue agent dialogue state tracker (dst) in a standard tds pipeline, the rule-based dst is deployed to keep track of information emerging in interactions between users and the dialogue agent. the output from the natural language understanding (nlu) module is fed to the dst to extract information, including informable slots about the constraints from users and requestable slots that indicate what users inquire about. in our setup, the dialogue agents and soa a 1 a 2 a 1 a 2 eoa gru gru gru mlp state v s v s v s figure 1: architecture to approximate a dialogue policy with an action decoder 2 . user-simulators are interacting through predefined dialogue actions therefore no nlu is involved. besides, a belief vector is maintained and updated for each slot in every domain. dialogue state we formulate a structured state representation s t according to the information resulting from the dst at time step t. there are 4 main types of information in the final representation: ## dialogue policy learning (pl) to fully make use of information contained in cooccurrence dependencies, we decompose the multilabel classification task in dialogue pl as follows. assuming the system response consists of two atomic actions, 'hotel-inform-address' and 'hotel-1 there are 166 atomic actions in total in the action space inform-phone', the model takes the dialogue state as input and predict the atomic actions sequentially. the path could be described as either 'hotel-informaddress' → 'hotel-inform-phone' or 'hotel-informphone' → 'hotel-inform-address'. before the training stage, the relative order of all the atomic actions will be predefined and fixed. following this solution, we apply a gru-based @xcite decoder to model the conditional dependency between the actions in one single turn as shown in figure 1 . the proposed model first extracts state features v s by feeding the raw state input s to an multilayer perceptron (mlp). in the next state, the state representation v s will be used as the initial hidden state h 0 of action decoder gru . to avoid information loss during decoding, the input to the action decoder is: the starting input input 0 is the concatenation of starting action soa and state representation v s . a t-1 denotes the dialogue action in the prediction path at time step t -1 and embedding(a) returns the action embedding of the given action a. in the next steps, actions will be generated consecutively according to: ## experimental setup multiwoz datasset @xcite is a multi-domain dialogue dataset with 7 distinct domains 5 , and 10, 438 dialogues. the main used scenario is a dialogue agent is trying to satisfy the tourists' demands such as booking a restaurant or recommending a hotel with specific requirements. each dialogue trajectory is decomposed into a set of state-action pairs with the same tds that is used for training. in total, we have 56, 700 dialogue state-action pairs in the training set, with 7, 300 in the validation set, and 7, 300 in the test set. baselines three types of baselines are explored: (b1): supervised learning, where the dialogue action selection task is regarded as a multi-label classification problem. (b2): reinforcement learning (rl), where the reward function is handcrafted and defined as follows: at the end of a dialogue, if the dialogue agent accomplishes the task within t turns, it will receive t * 2 as a reward; otherwise, it will receive -t as a penalty. t is the maximum number of turns in each dialogue; we set it to 40 in all experiments. furthermore, the dialogue agent will receive -1 as an intermediate reward during the dialogue to encourage shorter interactions. in our experiments, we used three methods, including: gp-mbcm @xcite , acer @xcite , ppo @xcite . (b3): adversarial learning, where dialogue agent is trained with a user simulator, we conduct comparisons with two methods: gail (ho and ermon, 2016) and gdpl @xcite . the dialogue agents in gail and gdpl are both ppo agents while these two methods have different reward models. we report the performance of aldm (liu and lane, 2018) for completeness. ## conclusion in this work, we proposed two supervised learning approaches and one adversarial learning method to train the dialogue policy for tdss without building user simulators. the proposed methods can achieve state-of-the-art performance suggested by existing approaches based on reinforcement learning (rl) and adversarial learning. however, we have demonstrated that our methods require fewer training efforts, namely the domain knowledge needed to design a user simulator and the intractable parameter tuning for rl or adversarial learning. our findings have questioned if the full potential of supervised learning for dialogue policy learning (pl) has been exerted and if rl methods have been used in the appropriate tds scenarios.