Attention Modulation for Zero-Shot Cross-Domain Dialogue State Tracking (2022)

Abstract: Dialog state tracking (DST) is a core step for task-oriented dialogue systems aiming to track the user’s current goal during a dialogue. Recently a special focus has been put on applying existing DST models to new domains, in other words performing zero-shot cross-domain transfer. While recent state-of-the-art models leverage large pre-trained language models, no work has been made on understanding and improving the results of first developed zero-shot models like SUMBT. In this paper, we thus propose to improve SUMBT zero-shot results on MultiWOZ by using attention modulation during inference. This method improves SUMBT zero-shot results significantly on two domains and does not worsen the initial performance with the great advantage of needing no additional training.

## introduction task-oriented dialogue systems aim to provide information and perform tasks requested by a user during a dialogue (e.g., booking a train ticket or finding a restaurant). as the dialogue progresses, the user may add some criteria or change its goal, so the system needs to track the current goal of the user at each dialogue turn for the dialogue to succeed. the associated task is called dialogue state tracking (dst) and consists, in its most studied form, in updating the slots mentioned by the user (see figure 1 ). state-of-the-art models rely on deep learning models. however, a highly desirable feature of dialogue systems is the ability to scale to new domains without retraining but by taking advantage of knowledge already acquired in previous domains. thus in this paper we study "leaveone-out" cross-domain zero-shot transfer. for each domain, a model is trained on dialogues that do not contain slots of the target domain and is then evaluated on dialogues containing slots of the target domain. user: in the center of the city please system: sure, where would you like it to be? user: i'm looking for a reasonably priced thai restaurant restaurant-food: thai restaurant-price range: moderate restaurant-food: thai restaurant-price range: moderate restaurant-area: center dialogue management 1 2 turn dialogue dialogue state figure 1: example of dialogue along with the dialogue state at each turn. zero-shot cross-domain transfer studies on dst are relatively recent and are mainly conducted on the multiwoz dataset @xcite foot_0 . such zero-shot learning was first applied to trade and sumbt models @xcite , where trade @xcite relies on an rnn and sumbt @xcite on the pretrained language model bert @xcite and an rnn. instead of building new architectures, recent state-of-the-art models leverage large generative pre-trained language models like gpt-2 @xcite or t5 @xcite , and work on the form of the input itself by incorporating slot descriptions @xcite , showing labeled examples @xcite , or considering a slot as a question @xcite . however, no further work has been conducted on understanding and improving the results of firstdeveloped models. thus in this paper we propose different architectural variants of sumbt and introduce attention modulation to improve crossdomain zero-shot results on multiwoz 2.0. bert sv bert sv bert multi-head attention normalization rnn ontology domain-slot e.g. [cls] restaurantfood [sep] system's and user's utterance e.g. [cls] what kind of food would you like? [sep] french food please [sep] values of slot e.g. [cls] none [sep], [cls] don't care [sep], [cls] french [sep] ## sumbt the main idea of sumbt is to match each slotname to a slot-value from an ontology given a dialogue turn (a system's and a user's utterance). the architecture of the model is illustrated in figure 2 2 . during inference, any domain/slot-name pair can be used as query input as long as the ontology contains the list of values associated with the domain/slot-name pair. trained sumbt models can thus be applied to new domains after updating the ontology, and the models can predict new slots never seen during training. we re-implemented our own version of sumbt and conducted zero-shot cross-domain experiments. transfer is measured by computing the joint goal accuracy (jga) only on the slots of the target domain. it consists of the percentage of turns from all dialogues where all targeted slots-names are associated with the correct slot-value. all experiments are run on 5 random seeds. in the first line of ## attention modulation motivated by previous observations, we propose a method called attention modulation to push the model to predict the slot-value none less frequently for unknown slots. specifically, this would apply when predicting the dialogue state of a dialogue turn that refers to an unknown domain. however, doing this could lead the model to predict any other value except the correct one. thus we also describe two variants of sumbt, aiming to take advantage of similarities that naturally exist between the slots of the different domains. we hypothesize that it would help the model to increase transfer between domains and that our method would be more effective on these variants. ## conclusion and future work in this paper we proposed different variants of sumbt and introduced attention modulation. this method successfully improves sumbt original cross-domain zero-shot results on the attraction and the hotel domains by respectively 6.26 and 2.58 points with the triple attention variant, while not needing any additional training and never worsening original results. for further work, we plan to analyze in detail the results and conduct additional experiments to understand better the impact of attention modulation on the different domains. for example, we plan to introduce a variable β in place of the value 0 in equation 5 to study how changing the value of β can affect evaluation results with modulation. we also plan to study the possibility of extending the attention modulation to other architectures.