Addressing Domain Changes in Task-oriented Conversational Agents through Dialogue Adaptation (2023)

Abstract: Recent task-oriented dialogue systems are trained on annotated dialogues, which, in turn, reflect certain domain information (e.g., restaurants or hotels in a given region). However, when such domain knowledge changes (e.g., new restaurants open), the initial dialogue model may become obsolete, decreasing the overall performance of the system. Through a number of experiments, we show, for instance, that adding 50% of new slot-values reduces of about 55% the dialogue state-tracker performance. In light of such evidence, we suggest that automatic adaptation of training dialogues is a valuable option for re-training obsolete models. We experimented with a dialogue adaptation approach based on fine-tuning a generative language model on domain changes, showing that a significant reduction of performance decrease can be obtained.

## introduction most of the recent approaches in task-oriented dialogue systems @xcite assume that each component is trained using annotated dialogues. such data-driven approaches are common both for intent detection and slot filling @xcite and for dialogue state tracking @xcite . in this paper, we deal with the situation where we have a conversational dataset, i.e., a collection of annotated dialogues for a certain domain (e.g., booking a restaurant), and then the domain changes (e.g., new restaurants open, or some restaurants change food or price). we investigate to what extent dialogue models trained on the initial dialogues are adequate for the occurred changes, and whether those dialogues can be automatically adapted to domain changes. being able to manage domain changes is highly relevant for practical purposes, as the process of data collection (in the order of several thousand dialogues for a medium-size domain) for an application domain is both rather complex (e.g., wizard of oz @xcite ) and very expensive. on the other side, automatic adaptation of training dialogues @xcite ) such that they reflect domain changes, is still a challenging research goal. in the paper, we first provide a definition for the domain changes we are interested in. second, we set an experimental framework, including evaluation metrics, where we can simulate how the performance of current dialogue models is sensitive to different kinds and amounts of domain changes. finally, a relevant contribution of the paper is the design and experimentation of unsupervised approaches for training dialogue models from synthetically adapted dialogues. to be more concrete, figure 1 presents an example of the situation we are addressing. on the left side we have an initial knowledge base (a) for the restaurant booking domain, and, below it, we show a reservation dialogue (c) between a user and a system. we assume that this dialogue has been human-generated (e.g., through wizard of oz) and that it is coherent with the content of the knowledge base. then, we assume that the knowledge base changes (reported on the right side of figure 1 (b)), as a new restaurant has opened offering a new kind of food, and one restaurant closed. according to these changes, the initial dialogue is no more consistent with the new domain. our intuition is that an adapted dialogue should preserve as much as possible the dialogic structure (e.g., user communicative goals, order of turns, conversation style) of the original dialogue, while it should be adapted to changes occurred in the domain knowledge. in principle, both the user requests and the system responses might be affected by domain changes. on one side the user might be (partially) aware of changes, and adapt its goals and requests (e.g., in figure 1 the user might be aware of a new restaurant with poke food, and ask about it). on the other side we assume that the system is fully aware of domain changes and that, in order to provide correct information to the user, its responses have to be coherent with such changes. in this context, dialogue adaptations are well defined changes of an original dialogue that make the dialogue coherent with a new domain knowledge. such adaptation include changing the name of a slot value (e.g., poke in place of british in figure 1 ), change number of instances (e.g., one in place of 3), change name of instances (e.g., poke house in place of the oak bistro). overall, the structure of the initial dialogue has been as much as possible preserved, while modifications aim at reflecting the occurred changes and reconstructing consistency with the new domain. although the long-term perspective of our research is to fully automatize dialogue adaptation, the goal of this paper is limited to an investigation of the main issues behind it, following the research directions mentioned above. particularly, we are interested in two aspects: first, assessing the impact of domain changes in current dialogue state tracking models; second taking advantage of the generative capacity of large pre-trained language models (e.g., @xcite @xcite ) to provide appropriate dialogue adaptations. more specifically, section 2 presents the relevant background on task-oriented dialogues, particularly on dialogue state tracking and dialogue manager. sections 3.1 and 3.2 face domain changes due to, respectively, slot-value and instance alterations, showing their impact on dialogue models. section 4 introduces dialogue adaptation, which is put into practice with some initial experiments presented in section 5 and corresponding results, which are discussed in section 6. finally, section 7 presents dialogue adaptation in the context of recent work in the field. ## background on task-oriented dialogues this section provides background on task-oriented dialogues, with a focus on how domain knowledge is represented, data-driven approaches, dialogue state tracking and dialogue manager. ## domain changes in this section we define a number of domain changes that will be investigated in the paper. ## dialogue adaptation in this section, we provide a definition for the dialogue adaptation task, as well as its main features. ## experiments we now define an experimental framework for simulating domain changes in a conversational system. we have two main goals: (i) investigate the impact of the domain changes defined in section 3 on a model trained on d 0 _train dialogues and tested on d 1 _test adapted dialogues; (ii) simulate the use of a model trained on adapted d 1 _train dialogues and tested on d 1 _test adapted dialogues. for the first goal, we consider both a dst model (for slot-value changes) and a dm model (for instance changes), while for the second goal we carry on a comparison on a dst model. ## results and discussion in this section we present the results obtained on the five experiments introduced in section 5. ## related work this section presents relevant work related to dialogue adaptation. although the idea of adapting dialogues to reflect domain changes is, to the best of our knowledge, original, there have been numerous attempts to modify or extend training data in order to make models more robust to unseen dialogue phenomena. delexicalization. the most similar approach to dialogue adaptation is delexicalization, which consists of substituting the slot-values in the dialogue with their corresponding slots (e.g., "i'm looking for italian food" with "i'm looking for food-type food"), or other placeholders @xcite . although the idea is that the dialogue model can generalize enough for recognising different slots for similar kbs, delexicalization, in practice, does not achieve good performance. on the multiwoz data-set, a trade dst model trained on delexicalized slot-values has a joint accuracy of 0.014, with a significant decrease in performance. ## conclusion dialogue adaptation is useful when collecting and annotating new dialogues for certain domain changes becomes too costly, and a cheap solution is preferable. the idea is that domain changes (e.g., new slot-values, new instances) are reflected in training dialogues through corresponding automatic adaptations. we have provided empirical evidence that current dialogue models, both dialogue state tracking and dialogue manager, are strongly affected by domain changes, with a significant decrease in performance. we discussed a number of issues that make automatic dialogue adaptation a challenging task. as for future work, the main goal is to improve dialogue adaptation techniques. while the use of pre-trained generative language models fine-tuned to the specific domain changes is promising, the amount of generated noise is still high, and more work is necessary to better constraint the slot-value generation to achieve humanlike performance.