Divide, Conquer, and Combine: Mixture of Semantic-Independent Experts for Zero-Shot Dialogue State Tracking (2023)

Abstract: Zero-shot transfer learning for Dialogue State Tracking (DST) helps to handle a variety of task-oriented dialogue domains without the cost of collecting in-domain data. Existing works mainly study common data- or model-level augmentation methods to enhance the generalization but fail to effectively decouple semantics of samples, limiting the zero-shot performance of DST. In this paper, we present a simple and effective “divide, conquer and combine” solution, which explicitly disentangles the semantics of seen data, and leverages the performance and robustness with the mixture-of-experts mechanism. Specifically, we divide the seen data into semantically independent subsets and train corresponding experts, the newly unseen samples are mapped and inferred with mixture-of-experts with our designed ensemble inference. Extensive experiments on MultiWOZ2.1 upon T5-Adapter show our schema significantly and consistently improves the zero-shot performance, achieving the SOTA on settings without external knowledge, with only 10M trainable parameters.

## introduction dialogue state tracking (dst) plays an important role in many task-oriented dialogue systems @xcite . the goal of this task is to understand users' needs and goals by exacting dialogue states at each turn, which are typically in the form of a list of slot-value pairs @xcite . accurate dst performance can help downstream applications such as dialogue management. however, collecting and annotating the dialogue state is notoriously hard and expensive @xcite . this problem becomes i would like a taxi from saint johns college to pizza hut fen ditton. ## related work dialogue state tracking (dst) has been of broad interest to the dialogue research community. existing dst models require plenty of state labels @xcite @xcite , which is hard to get in real scenarios. various studies on dst with zero-shot learning have been conducted to tackle unseen slots @xcite from the data or model perspective. firstly, data augmentation is widely used to improve the effectiveness of the existing dst models. @xcite synthesizes dialogues for a new domain using domain templates derived from observing a small dataset and the ontology of the domain. other studies utilize diverse labeled datasets from other tasks, such as dialogue summarization task @xcite or generative question answering task @xcite , also called zero-shot cross-task transfer. in this paper, we focus on zero-shot cross-domain dst, where the model is first trained on several domains and transferred into unknown domains. many works focus on developing the advantage model or framework to enhance the robustness of dst @xcite @xcite . @xcite adopts the bert to produce context representations of dialogue context and applies span prediction modules to predict the slot value as a text span. @xcite encodes the whole dialogue context and decodes the value for every slot using a copy-augmented decoder. recently, many pre-trained language models, such as gpt @xcite and t5 @xcite , demonstrate impressive zero-shot learning ability and attract many researchers. @xcite proposes to model multi-dataset question answering with a collection of single-dataset experts -dataset-specific adapter modules @xcite . in dst, @xcite first leverages the slot description as a prompt and generates the slot value for zero-shot cross-domain settings. @xcite models three types of slot dependency based on prompt learning and further improves the zero-shot performance. but these approaches mainly benefit from the similarity across slots and language knowledge inside pretrained models, ignoring the different semantics areas of seen data and failing to the effective inference on unseen domains. ## background notation. we define {(a 1 , u 1 ), .., (a t , u t )} as a set of utterances from two speakers, where a and u represent the system response and user utterance, respectively. at turn t, we denote the dialogue context as c @xmath0, u 1 ), . . . , (a t , u t )}, which includes t turns from system and user. the task of dst is to predict the dialogue state b t given dialogue context c t . the dialogue state, b t , is represented as slot-value pairs, denoted as b @xmath1, v 1 ), . . . , (s j , v j )} where s j and v j denote the j-th slot name and value at turn t. j is the total number of slots in all domains. generation-based dst. unifying the dialogue states tracking as generation task shows promising performance, where it follows an auto-regressive fashion @xcite . for each turn, a pre-trained language model (e.g t5) takes the dialogue context c t and the slot name s j as input and decodes the corresponding slot value v j . the objective l is to minimize the negative log-likelihood loss on all slots: ## experiments dataset we evaluate our method on widely-used multi-domain datasets multiwoz @xcite and schema-guided dataset @xcite . the multiwoz dataset contains 10k+ dialogues across 7 domains. each dialogue consists of one or multiple domains. we follow the previous pre-processing and evaluation setup @xcite , where the restaurant, train, attraction, hotel, and taxi domains are used for zero-shot cross-domain experiments. the schema-guided dialogue (sgd) dataset consists of over 16k+ multi-domain dialogues and covers 16 domains. the test set contains unseen data to measure the performance in the zero-shot setting. detailed data statistics are shown in appendix a. evaluation metrics we follow @xcite to use slot accuracy (sa) and joint goal accuracy (jga) as evaluation metrics. sa is calculated as the ratio of individual slot in which its value is correctly predicted, and jga measures the percentage of correct in all dialogue turns, where a turn is considered as correct if and only if all the slot values are correctly predicted. in zero-shot dst @xcite , the model obtains all training data from the training dialogues except for an unseen domain, which is used to evaluate. pre-trained on qa datasets and then applied to unseen domains. t5dst @xcite ) explores the slot description as a prompt to generate slot values. slotdm-dst @xcite models three types of slot dependency, i.e., slot-slot, slot-value, and slot-context, to improve zero-shot dst. sgd-baseline utilizes schema descriptions to predict the dialogue state of unseen domains. moreover, we implement t5-adapter that concatenates the dialogue context and slot name as inputs, following t5dst, as the fair baseline of our method. different from other baselines finetuning all parameters, t5-adapter only tunes the parameters of the adapter during training. all baselines listed here do not consider any information from new domains. for a fair comparison, we don't include the in-context learning work on @xcite because they design specific prompts using the information from the unseen domain. implementation our models are implemented in pytorch (paszke et al., 2019) using huggingface @xcite and the adapter-transformers library @xcite . in division processing, we utilize t5-base @xcite as the context encoder and apply mean pooling on the outputs of the encoder as the dialogue vectors. we choose kmeans (hartigan and wong, 1979) as the clustering algorithm and set the number of sub-sets as 3. in conquer processing, t5 is employed as the dst expert with the default adapter configuration from @xcite foot_0 , which adds approximately 0.8m parameters to the t5-small (60m) and 3.6m parameters to the t5-base (220m). we freeze the transformer parameters and use a learning rate of 1e-4 on adapter parameters for each expert. for all experiments, we train each independent expert for 10 epochs. we use the adamw optimizer @xcite and set the batch size to 16. in the combining process, the scale temperatures are set to 2 and 0.2 in the token-and parameter-level ensemble inference, respectively. for a fair comparison, we process and evaluate the multiwoz datasets following t5dst @xcite . in the sgd dataset, we process the data following transferqa @xcite and use the official evaluation script foot_1 to evaluate. ## discussion to better understand our proposed schema, we first present essential ablation studies in §6.1, and show in-depth analyses on clustering ( §6.2) and ensemble inference ( §6.3), respectively. additionally, we discuss the complementarity of our framework with others in §6.4. ## conclusion in this paper, we propose a new learning schema "divide, conquer, and combine" to improve the zeroshot generalization in dst. the philosophy behind this is to explicitly divide the seen data into different semantic areas, such disentanglement provides flexibility for mapping the unseen sample to the different experts trained on corresponding semantic areas, and the ensemble results of experts are expected to improve the model generalization. the experimental results indicate that our model using small trainable parameters reaches state-of-art performances in zero-shot cross-domain dst.