Addressing Slot-Value Changes in Task-oriented Dialogue Systems through Dialogue Domain Adaptation (2021)

Abstract: Recent task-oriented dialogue systems learn a model from annotated dialogues, and such dialogues are in turn collected and annotated so that they are consistent with certain domain knowledge. However, in real scenarios, domain knowledge is subject to frequent changes, and initial training dialogues may soon become obsolete, resulting in a significant decrease in the model performance. In this paper, we investigate the relationship between training dialogues and domain knowledge, and propose Dialogue Domain Adaptation, a methodology aiming at adapting initial training dialogues to changes intervened in the domain knowledge. We focus on slot-value changes (e.g., when new slot values are available to describe domain entities) and define an experimental setting for dialogue domain adaptation. First, we show that current state-of-the-art models for dialogue state tracking are still poorly robust to slot-value changes of the domain knowledge. Then, we compare different domain adaptation strategies, showing that simple techniques are effective to reduce the gap between training dialogues and domain knowledge.

## introduction conversational agents are receiving great attention in recent years, both in research and applications @xcite , mainly because of the progress achieved by neural approaches in modeling dialogue phenomena @xcite , @xcite . particularly, we focus on task-oriented dialogue systems @xcite , which are able to assist a user for specific tasks (e.g., booking a restaurant, taking an appointment, execute commands) in a certain conversational domain. such data-driven dialogue systems typically learn a model from annotated training dialogues (e.g., @xcite , @xcite , @xcite ), and such dialogues, in turn, are collected and annotated according to a certain domain scenario (e.g., restaurants in a certain town, songs in a certain music data-set, etc.). once the model is trained, it is applied to understand new conversations in the same domain, or in a similar domain. however, in real scenarios, domain knowledge is subject to frequent changes, and initial training dialogues may soon become obsolete, resulting in a significant decrease in the model performance. it is common that new domains (e.g., restaurant) are added or removed for a certain conversational scenario, as well as slots, slot-values and instances. such situations require the capacity of the dialogue system to adapt its behaviours to domain knowledge changes. depending on the complexity of the changes that occur, domain adaptation of data-driven systems can be approached in two directions: (i) improving the model robustness, and (ii) adapting the training dialogues to the new situation. while the first direction has been largely explored through several techniques, including transfer learning @xcite , and zero-shot learning though schema-guided models @xcite @xcite @xcite , and delexicalization @xcite , in this paper we take the second, less investigated, perspective, focusing on the relation between training dialogues and domain knowledge. we have defined a new experimental setting for dialogue domain adaptation, where, given an initial conversational domain (kb-source), available training dialogues (d-source) are adapted to be as much as possible consistent to a modified knowledge base (kb-target), resulting in new set of dialogues (d-target). then, we use a state-of-the-art model for dialogue state tracking and assess the performance of different adaptation strategies against a gold standard of manually adapted target dialogues. we run a number of experiments showing that: (i) current state-of-the-art models are very poorly robust to changes over the domain knowledge; finally, as part of our study, we highlight that current component-based evaluation settings for task-oriented dialogue systems (i.e., slot filling, intent detection, dialogue state tracking, utterance generation) are not sensible to correctness of system responses, which, instead, is crucial to assess domain adaptability. the paper is structured as follows. section 2 provides basic background in task-oriented dialogue systems. section 3 defines the conversational adaptation task, while in section 4 we introduce the proposed dialogue domain adaptation approach. section 5 introduces the new experimental setting, and section 6 provides the results of our experiments and discusses them. ## task-oriented dialogue framework this section provides background related to taskoriented dialogue systems, particularly about how domain knowledge is managed and about dialogue state tracking. ## dialogue domain adaptation: task definition in this section we define dialogue domain adaptation (dda) and its core properties. in our setting we assume an initial conversational domain, represented in a kb-source, and corresponding annotated training dialogues d-source. then, as in real application scenarios, we assume that a number of changes occur in kb-source, such that a new conversational domain kb-target needs to be considered. dialogue domain adaptation consists in the capacity to automatically produce new annotated dialogues d-target, such that they maintain both the linguistic structure and the linguistic variability of the initial d-source dialogues, while, at the same time, being consistent with the new kb-target. figure 1 provides a concrete example of dda. here we have a source dialogue, taken from the multiwoz data-set of restaurant booking conversations, mentioning restaurants in a certain region. notice that it is implicitly assumed that the system responses are true facts in a kb-source (e.g., if the system says what about eraina?, it means that kb-source contains a restaurant named eraina). however, this might not be true in kb-target, where the eraina restaurant might not exist anymore. the target dialogue in figure 1 is basically the same dialogue as the source dialogue, although adapted to be consistent with a kb-target. dda focuses on the automatic generation of such d-target dialogues starting from d-source dialogues. in the rest of the section we consider three core characteristics that affect dda: domain changes, dialogue internal coherence, and kb-dialogue adherence. ## substitution-based dda we approach the dialog domain adaptation task described in section 4 through the substitution of slot-values in d-source with slot-values selected from kb-target. figure 2 depicts the elements of our experimental setting, highlighting the relationships between them. the dataset d-source consists of both training and test dialogues, with the latter possibly containing a certain number of slotvalues that are unseen in the training set. d-source dialogues are collected in a strong connection with kb-source, which has been quantified through a kb-dialogue adherence measure. given a certain kb-target, which differs from kb-source for a proportion of slot-values estimated by the kb-overlap, the kb mapping defines the substitutions that need to be done for every slot-value, and, on the basis of this mapping, the adaptation process generates the d-target data-set. ## experimental setting this section describes the experiments that we carried on to test dialogue domain adaptation based on slot-value substitutions. the experimental setting includes an initial kb-source and corresponding d-source dialogues; a set of handcrafted test d-source dialogues; few substitution algorithms that we experimented to produce different d-target dialogues; and a state-of-art dialogue state tracker to check the performance of different adaptation strategies. ## results and discussion the remaining experiments were ran on three different d-target datasets, created on the basis of different strategies, as explained in section 5.4. random-d was made by substituting every source slot-value with a slot-value that was taken randomly from a list of all unique slot-values that are present in the dialogue. this means that for every substitution, there was the same chance of picking a very frequent value -such as "indian food" -than picking a value that occurs only once -such as "south caribbean spicy food". for this reason the joint goal accuracy is very low, even if significantly better than in the no adaptation setting. a major improvement, however, happens when we use data-sets that are based on an adapted strategies based on the kb. the random-kb strategy, in fact, has a joint goal accuracy slightly lower than the original cam experiment, while the frequency-based strategy even exceeds cam with an goal accuracy over 50%. the difference in performance between these last two data-sets can be explained by considering that with random-kb we randomly changed the assignment of the slot-values to be substituted once for every dialogue, which can be beneficial for the model capability of generalizing, but that does not allow it to maximize the ## conclusion domain knowledge in conversational agents is subject to frequent changes, and this leads to the necessity of continuously updating training dialogues in order to keep them consistent with domain knowledge. as collecting conversational dialogues by hand requires a significant effort, approaches for automatically updating are required. in this paper we have proposed dialogue domain adaptation, a methodology for operating changes to an initial training dialogue, so that it becomes adherent to a modified domain knowledge. the experiments that we conducted reveal a twofold evidence: they demonstrate that zero adaptation results in a significant loss in dst performance, and they show that simple substitution-based adaptation methods bring instead effective results. moreover, the experiments on different adaptation methods showed diverse phenomena. while the best performance is obtained using a frequency strategy -which maps the most frequent slotvalue of the source domain to the most frequent slot-value of the target domain -a random strategy based on kb values performed slightly worse, and a severe drop in the results occurred when using a random strategy based on dialogue values. linguistic variability is perhaps an important factor that emphasises this difference, and it will be an interesting topic to be explored in further works.