Learning to Learn End-to-End Goal-Oriented Dialog From Related Dialog Tasks (2021)

Abstract: For each goal-oriented dialog task of interest, large amounts of data need to be collected for end-to-end learning of a neural dialog system. Collecting that data is a costly and time-consuming process. Instead, we show that we can use only a small amount of data, supplemented with data from a related dialog task. Naively learning from related data fails to improve performance as the related data can be inconsistent with the target task. We describe a meta-learning based method that selectively learns from the related dialog task data. Our approach leads to significant accuracy improvements in an example dialog task.

## introduction one key benefit of goal-oriented dialog systems that are trained end-to-end is that they only require examples of dialog for training. avoiding the modular structure of pipeline methods removes the human effort involved in creating intermediate annotations for data to train the modules. the endto-end structure also enables automatic adaptation of the system, with different components of the model changing together. this flexibility is particularly valuable when applying the system to a new domain. however, end-to-end systems currently require significantly more data, increasing the human effort in data collection. the most common method for training is supervised learning (sl) using a dataset of dialogs of human agents performing the task of interest @xcite @xcite . to produce an effective model, the dataset needs to be large, high quality, and in the target domain. that means for each new dialog task of interest large amounts of new data has to be collected. the time and money involved in that collection process limits the potential application of these systems. we propose a way to reduce this cost by selectively learning from data from related dialog tasks: tasks that have parts/subtasks that are similar to the new task of interest. specifically, we describe a method for learning which related task examples to learn from. our approach uses meta-gradients to automatically meta-learn a scalar @xmath0, 1) for each of the related task data points, such that learning from the weighted related task data points improves the performance of the dialog system on the new task of interest. these weights are dynamically adjusted over the course of training in order to learn most effectively. we still learn from data for the target task, but do not need as much to achieve the same results. to demonstrate this idea, we considered two experiments. first, we confirmed that the method can work in an ideal setting. we constructed a classification task where the related task data is actually from the same task, but with the incorrect label for 75% of examples, and there is an input feature that indicates whether the label is correct or not. our approach is able to learn to ignore the misleading data, achieving close to the performance of a model trained only on the correct examples. second, we evaluated the approach on a personalized restaurant reservation task with limited training data. here, the related task is also restaurant reservation, but without personalization and with additional types of interactions. we compare our approach to several standard alternatives, including multi-task learning and using the related data for pre-training only. our approach is consistently the best, indicating its potential to effectively learn which parts of the related data to learn from and which to ignore. successfully learning from available related task data can allow us to build end-to-end goal-oriented dialog systems for new tasks faster with reduced cost and human effort in data collection. ## related work the large cost of collecting data for every new dialog task has been widely acknowledged, motivating a range of efforts. one approach is to transfer knowledge from other data to cope with limited availability of training dialog data for the new task of interest. for example @xcite split the dialog model such that most of the model can be learned using ungrounded dialogs and plain text. only a small part of the dialog model with a small number of parameters is trained with the dialog data available for the task of interest. in contrast, we explore how to learn from related grounded dialogs, and also without any specific constraints on the structure of the end-to-end dialog system architecture. @xcite pre-train the model with data automatically generated from different tasks and @xcite use pre-trained language models as initialization and then fine-tune the dialog model with data from the task of interest. these ideas are complementary to our approach as we make no assumptions about how the model was pre-trained. recently, there has been work that explored ways to automatically learn certain aspects of the transfer process using meta-learning. @xcite look at the problem of learning a joint dialog policy using reinforcement learning (rl) in a multi-domain setting which can then be transferred to a new domain. they decomposed the state and action representation into features that correspond to low level components that are shared across domains, facilitating cross-domain transfer. they also proposed a model agnostic meta learning @xcite based extension that learns to adapt faster to a new domain. madotto et al. (2019), mi et al. (2019), qian and yu (2019) and dai et al. (2020) also look at multi-domain settings. they use maml based meta-learning methods to learn an initialization that adapts fast with few dialog samples from a new task. all of the papers above consider settings where there is access to a large set of training tasks. the meta-learning systems learn to transfer knowledge to a new test task by learning how to do transfer on different training tasks. while each task only has a limited amount of dialog data, they need a lot of tasks during training. in contrast, we look at a setting where the task from which we want to transfer knowledge from and the task that we want to transfer knowledge to are the only tasks that we have access to at training time. any learning about how to transfer knowledge has to happen from just these two tasks. none of the above methods are applicable to this setting. learning a task, while simultaneously metalearning certain aspects of the learning process has been done successfully in some sl and rl settings recently. @xcite ; @xcite use meta-learning to adapt hyperparemeters such as learning rate and and even learn entire optimizers themselves during training for sl tasks such as image classification. given a single task, @xcite successfully meta-learn intrinsic rewards that help the agent perform well on that task. @xcite use meta-gradients to learn rl training hyperparameters such as the discount factor and bootstrapping parameters. the metagradient technique used in our proposed method is closely related to @xcite . they learn intrinsic rewards for an rl agent acting in given domain, such that learning with those intrinsic rewards improves the performance of the agent in the task of interest in a different domain. while we use a meta-learning based method for learning the weights for the related task data points in this work, there are other techniques in the machine learning literature, especially in the computer vision literature, that can potentially be used to learn the weights. a large section of these recent techniques are based on learning an adversarially trained discriminator for estimating the weights of related image classification task data points @xcite @xcite . @xcite use a combination of several domain adaptation heuristics to assign weights and evaluate on nlp tasks. @xcite cluster the related task data points and learn attention weights for the clusters. an interesting future direction would be to study which weighting methods are best suited for end-to-end learning of neural goaloriented dialog systems using related tasks and under what conditions. ## experiments and results we first illustrate our proposed method on a simple image classification task with a hand designed related task that allows us to verify if the proposed method can learn meaningful weights. we then evaluate our proposed method on the task of personalized restaurant reservation. ## conclusion end-to-end learning of neural goal-oriented dialog systems requires large amounts of data for training. collecting data is a costly and time consuming process. in this work we showed on an example dialog task we can utilise a related task's data to improve the performance of a new task of interest with only a limited amount of data. our proposed method uses meta-learning to automatically learn which of the related task data points to selectively learn from. an important future work is to evaluate/extend the proposed method to more challenging and complex dialog datasets/tasks. data useful for a dialog task of interest (related data) could be present in different formats. the related data could include, for example, natural language instructions on how to perform the task of interest, or a description of how the new dialog task of interest is different from a related dialog task. an interesting future direction is to investigate methods to successfully utilise such related data. @xcite