Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain Dialogue State Tracking (2020)

Abstract: Zero-shot transfer learning for multi-domain dialogue state tracking can allow us to handle new domains without incurring the high cost of data acquisition. This paper proposes new zero-short transfer learning technique for dialogue state tracking where the in-domain training data are all synthesized from an abstract dialogue model and the ontology of the domain. We show that data augmentation through synthesized data can improve the accuracy of zero-shot learning for both the TRADE model and the BERT-based SUMBT model on the MultiWOZ 2.1 dataset. We show training with only synthesized in-domain data on the SUMBT model can reach about 2/3 of the accuracy obtained with the full training dataset. We improve the zero-shot learning state of the art on average across domains by 21%.

## introduction automated conversational agents can reduce the costs of customer support, a necessary service in just about every business. however, training a goal-directed dialogue agent for a domain often requires acquiring annotated dialogues to cover all possible conversation flows. commonly, this is done using the wizard-of-oz technique @xcite , where two crowdsource workers converse with each other, while also annotating the state at each turn. this technique has been employed to construct several datasets @xcite @xcite . recently, it has been used to build the multiwoz dataset @xcite , a large corpus of dialogues across 7 domains. unfortunately, not only is the initial acquisition expensive, annotating dialogues correctly has proven to be challenging due to human errors, delays in annotation, inconsistent conventions, and normalization issues @xcite s: start u: can you help with information regarding a food place? i need to book at 15:45. s: searchrequest restaurant(book @xmath0:45") a: how about the restaurant with name la tasca and italian food? u: can you find something which serves seafood? s: searchrequest restaurant(book @xmath1:45", @xmath2: what date are you looking for? u: thursday please. s: searchrequest restaurant(book @xmath3:45", @xmath4, book @xmath5: how about the copper kettle? it is a food place with seafood food. u: what is the price range and the area? s: slotquestion restaurant(book @xmath6:45", @xmath7, book @xmath8, price @xmath9, @xmath10: the copper kettle is a moderately priced restaurant in the north of the city. would you like a reservation? u: no, thanks. s: close restaurant(book @xmath11:45", @xmath12, book @xmath13: can i help with you anything else? u: thank you, that will be it for now. s: end restaurant(book @xmath14:45", @xmath15, book @xmath16: an example of a dialogue that can be synthesized from our templates. 'u:' indicates the user, 'a:' the agent, and 's:' is the dialogue state at each turn. small, 2019). the multiwoz dataset still has significant inconsistencies @xcite despite having been constructed through multiple rounds of annotations @xcite . we observe empirically from the multiwoz training data that conversations in all the domains follow the same pattern: the agent and user start by greeting each other, tehn they converse to find a proposal that satisfies the user, the user provides additional required information, and finally the agent completes the user's transaction. to facilitate transfer learning, we create an abstract model of dialogues that is independent of the domain of the conversation. in this paper we will focus on dialogues for transactions; other kinds of dialogues such as opinion sharing will have different models. we have developed an algorithm that accepts an ontology of a domain and a few phrases commonly used in that domain. the algorithm synthesizes dialogue training data based on an abstract dialogue model. the dialogue synthesized consists of turns of conversation, each of which has a start state, an agent utterance, a user utterance, and an end state. the start and end states summarize the semantics of the conversation at those points. an example of a dialogue that can be synthesized by our model is shown in fig. 1 . to transfer knowledge to a new domain in a zero-shot setting, we train with the synthesized data for the new domain together with existing data for other domains. in addition, we adapt training samples from related domains by substituting them with the vocabulary of the new domain. we can improve the accuracy of the abstract dialogue model as well as the state-tracking neural network by iteratively refining the model based on the error analysis on the validation data, and by introducing additional annotations in the new domain. note that the abstract dialogue model can be also used directly to implement the agent itself. the contributions of this paper are as follows: the bert-based sumbt model can, in a purely zero-shot fashion, achieve between 61% and 92% of the accuracy obtained by a model trained on the full dataset. we propose combining pretrained models with synthesized data as a general technique to bootstrap new dialogue state trackers. more recently, we proposed training with both a large amount of synthesized data and a small amount of paraphrase data for semantic parsing of single sentences @xcite . we showed that training with such data can perform well on real-world evaluations. this paper extends this work to the multi-turn setting. dialogues are more complex as they need to capture information, such as the abstract dialogue state, that is not present in the target annotation (domain and slot values). we extend the synthesis algorithm to operate based on a dialogue model, tracking enough information to continue the dialogue. we also present a novel dialogue model that is suitable for synthesis. ## dialogue-model based synthesis in this section, we first define abstract dialogue models, then describe how we can generate dialogues based on the model. we also describe the techniques we use to adapt training dialogues from other domains to the new domain. ## conclusion we propose a method to synthesize dialogues for a new domain using an abstract dialogue model, combined with a small number of domain templates derived from observing a small dataset. for transaction dialogues, our technique can bootstrap new domains with less than 100 templates per domain, which can be built in a few person-hours. with this little effort, it is already possible to achieve about 2/3 of the accuracy obtained with a large-scale human annotated dataset. furthermore, this method is general and can be extended to dialogue state tracking beyond transactions, by building new dialogue models. we show improvements in joint accuracy in zero-shot and few-shot transfer learning for both the trade and bert-based sumbt models. our technique using the sumbt model improves the zero-shot state of the art by 21% on average across the different domains. this suggests that pretraining complements the use of synthesized data to learn the domain, and can be a general technique to bootstrap new dialogue systems. we have released our algorithm and dialogue model as part of the open-source genie toolkit, which is available on github foot_3 .