Amalgamating Knowledge from Two Teachers for Task-oriented Dialogue System with Adversarial Training (2020)

Abstract: The challenge of both achieving task completion by querying the knowledge base and generating human-like responses for task-oriented dialogue systems is attracting increasing research attention. In this paper, we propose a “Two-Teacher One-Student” learning framework (TTOS) for task-oriented dialogue, with the goal of retrieving accurate KB entities and generating human-like responses simultaneously. TTOS amalgamates knowledge from two teacher networks that together provide comprehensive guidance to build a high-quality task-oriented dialogue system (student network). Each teacher network is trained via reinforcement learning with a goal-specific reward, which can be viewed as an expert towards the goal and transfers the professional characteristic to the student network. Instead of adopting the classic student-teacher learning of forcing the output of a student network to exactly mimic the soft targets produced by the teacher networks, we introduce two discriminators as in generative adversarial network (GAN) to transfer knowledge from two teachers to the student. The usage of discriminators relaxes the rigid coupling between the student and teachers. Extensive experiments on two benchmark datasets (i.e., CamRest and In-Car Assistant) demonstrate that TTOS significantly outperforms baseline methods.

## introduction task-oriented dialogue systems (tdss), which help users to complete specific tasks with natural language, have attracted increasing attention recently due to the broad applications such as event scheduling and flight booking. conventional tdss have * min yang is corresponding author a complex pipeline @xcite , which consists of modularly connected components for natural language understanding (nlu), dialogue state tracking (dst), and dialogue policy (dp). a limitation of such pipelined design is that errors made in upper stream modules may propagate to downstream components, making it hard to identify and track the source of errors. in addition, these methods usually require a large number of handcrafted features and labels, which may restrict the expressive power and learnability of the models. to ameliorate the limitations with the conventional pipeline tdss, great efforts have been made in designing deep neural network-based end-to-end solutions @xcite @xcite . recent advances are overwhelmingly contributed by sequence-to-sequence (seq2seq) models @xcite @xcite , which have taken the state-of-the-art of tdss to a new level. these methods map dialogue context to output responses directly without explicitly providing handcrafted features and nlu/dst/dp labels, thus reduce human effort and are easily adapted to new domains. despite the effectiveness of previous studies, there are several technical challenges in building a tds that is capable of retrieving accurate entries from the knowledge base (kb) and generating human-like responses. (1) previous work @xcite shows that users of tdss tend to use succinct language which often omits entities or concepts made in previous utterances. however, seq2seq models often ignore how the conversation evolves as information progresses @xcite and thus result in generating incoherent and ungrammatical responses that are dominated by words appearing with high frequency in the training data. (2) seq2seq models suffer from effectively reasoning over and incorporating kb information @xcite . it is difficult to encode and decode the knowledge from a large and dynamic kb, making the response generation unstable. in addition, typically, a shared memory is used for both dialogue context and kb triples, making the tdss struggle to reason over the two forms of data. although some previous methods @xcite leverage separate memories for modeling dialogue context and kb facts, they either focus on capturing the dialogue patterns or retrieving accurate kb entities, but not both. one possible solution to the aforementioned problems is to explicitly encourage the seq2seq model to learn dialogue patterns and model the exterior kb knowledge retrieval with separate guidance for each. in this study, we propose a "two-teacher one-student" learning framework (ttos) for building a high-quality tds (student), where a student network is encouraged to integrate the knowledge from two expert teacher networks. concretely, a kb-oriented teacher network (t kb ) is trained via reinforcement learning with entity score as the reward, which specializes in retrieving accurate kb entities; a dialogue pattern-oriented teacher network (t dp ) is trained via reinforcement learning with bleu as the reward, which is expected to learn the language patterns of the dialogue @xcite , and thus specializes in generating coherent and grammatical responses. afterwards, we optimize the student with distilled expert knowledge from two teacher networks. our motivation is that the two teachers can provide different supervisory information that can be fully utilized through collaborative training. instead of adopting the classic student-teacher learning strategy of forcing the output of a student network to exactly mimic the soft targets produced by the teacher networks, we employ the generative adversarial network (gan) to transfer knowledge from two teachers to the student. to be more specific, the generator is the student network to produce dialogue responses, and the two discriminators distinguish the learned output representations from the student and teacher networks. by employing the output of the two discriminators as feedback, the student network can achieve collective success in both retrieving accurate kb entities and generating natural responses. this paper has three main contributions listed as follows. ## our methodology given the dialogue context @xmath0, x 2 , . . . , x m } with m words and the system response @xmath1, y 2 , . . . , y t } with t words, the dialogue system aims to optimize the generation probability of y conditioned on x, i.e., p(y|x). as illustrated in figure 1 , ttos consists of three networks: a kb-oriented teacher network (t kb ) that is specialized for retrieving entities from kb, a dialogue pattern-oriented teacher network (t dp ) that is specialized for learning the dialogue patterns, and a student network (s) that tries to extract accurate kb entities and generate human-like responses. the three networks share the same network structure but different training strategies. the learning procedure of ttos contains two stages of training. in the first stage, the three networks are pre-trained independently with different training strategies. in particular, the student network is trained with supervised learning, while the two teacher networks t kb and t dp are trained via the reinforcement learning (rl) with goal-specific rewards (i.e., entity score and bleu respectively), which can be viewed as experts towards the goals. then, we employ gan to learn the student network, where the generator is the student network to produce dialogue responses, and two discriminators distinguish the learned output responses from student and teacher networks. next, we will introduce the three networks and the gan-based student-teacher learning paradigm in detail. ## experimental setup dataset we evaluate the proposed ttos model on two widely used multi-turn task-oriented dialogue datasets: camrest @xcite and in-car assistant @xcite . the camrest dataset is composed of 676 human-tohuman multi-turn conversations in the restaurant reservation domain. the average number of turns per dialogue is about 5. following in @xcite , we divide the dataset into training/validation/testing sets with 406/135/135 dialogues respectively. the in-car assistant dataset contains 3031 multi-turn dialogues, which are divided into 2425/302/304 dialogues for training/validation/testing, respectively. in-car assistant includes three distinct domains: calendar scheduling, weather information retrieval, and point-of-interest navigation. there are 2.6 turns on average per dialogue. compared to camrest, the in-car assistant dataset is more diverse in the utterances, and the kb information is also more complicated. training details the grid search algorithm @xcite is applied on the validation set to automatically tune the hyper-parameters. we use the 300-dimensional word2vec vectors @xcite to 256. the number of hops for the memory network is set to 3. the recurrent weight parameters are initialized as orthogonal matrices. we initialize the other weight parameters with the normal distribution n (0, 0.01) and set the bias terms to zero. to stabilize the process of training gan, we use adam optimizer (kingma and ba, 2014) with a relatively small initial learning rate of 1e -4 to train the model. the batch size is set to 8. the step ratio of g and d is set to 1:1 for reaching a training balance. we set the value of α to 1.0, because a too large α value will make student rely excessively on teachers' outputs without concrete guidance at each time step, while a loss with too low α value cannot guide the student to fool the two discriminators, which may make the adversarial training process unstable. in addition, we also apply dropout (dropout @xmath2.2) on several layers of generator, but not for discriminators. it is noteworthy that we first pre-train the three networks separately by optimizing the networks with different training strategies. then, we switch to the gan training to learn the student network by amalgamating knowledge from the two teacher networks. ## conclusion in this paper, we propose a novel "two-teacher one-student" learning framework (ttos) for taskoriented dialogue, which aims to improve the performance of the task-oriented dialogue system in retrieving accurate entries from kb and generating human-like responses simultaneously. with adversarial learning, we train the student network to amalgamate expert knowledge naturally from the two teacher networks for the above two goals. the experimental results on two benchmark datasets demonstrated that our model achieves impressive results compared to the state-of-the-art task-oriented dialogue systems.