U ni DU : Towards A Unified Generative Dialogue Understanding Framework (2022)

Abstract: With the development of pre-trained language models, remarkable success has been witnessed in dialogue understanding (DU). However, current DU approaches usually employ independent models for each distinct DU task, without considering shared knowledge across different DU tasks. In this paper, we propose a unified generative dialogue understanding framework, named UniDU, to achieve effective information exchange across diverse DU tasks. Here, we reformulate all DU tasks into a unified prompt-based generative model paradigm. More importantly, a novel model-agnostic multi-task training strategy (MATS) is introduced to dynamically adapt the weights of diverse tasks for best knowlege sharing during training, based on the nature and available data of each task. Experiments on ten DU datasets covering five fundamental DU tasks show that the proposed UniDU framework largely outperforms task-specific well-designed methods on all tasks. MATS also reveals the knowledge sharing structure of these tasks. Finally, UniDU obtains promising performance on unseen dialogue domain, showing great potential of generalization.

## introduction the development of the conversational system plays an important role in the spread of the intelligence devices, such as intelligence assistants and car play. in recent years, there has been a growing interest in neural dialogue system @xcite @xcite @xcite @xcite @xcite @xcite . dialogue understanding is a core technology and hot topic in the dialogue system, aiming to analyze a dialogue from different fine-grained angles accurately. ## unidu in this section, we first introduce the unified sequence-to-sequence data format for the five du tasks. then we introduce the formulation of each task in detail, especially how to reformulate the intent detection, slot filling and dialogue state tracking as the generation task. there are three components in the input of unidu: task identification, dialogue content, and task query. the task identification represents with a special token, e.g., dialogue summary identified by "[ds]". the dialogue content means the taskdependent input, such as dialogue history for dialogue summary. the task query can be regarded as the task-specific prompt, which includes the task definition and domain-related information. there ds] emma: buy me some earplugs please [t] paul: how many pairs? [t] emma: 4 or 5 packs [t] paul: i'll get you 5 [t] emma: thanks [c] what is the summary of this dialogue? [dc] anna politkovskaya [t] the murder remains unsolved, 2016 [t] did they have any clues? [c] what is the semantic completion statement of 'did they have any clues?'? [id] what can i do if my card still hasn't arrived after 2 weeks? [c] what is the user's intent on the bank business? [sf] i am lakesha mocher [c] what is last name in general domain? [dst] i am looking for a place to to stay that has cheap price range it should be in a type of hotel [c] what is the user's constraint about the price range of the hotel? [ds] maya will buy 5 packs of earplugs for randolph at the pharmacy. [dc] did investigators have any clues in the unresolved murder of anna politkovskaya? [id] card arrival [sf] mocher [dst] cheap update mtl training strategy task identification dialogue content task query task identification query answer unidu figure 1: overview of unidu. under unidu framework, the input consists of three parts: task identification, dialogue content and task query, where ⊕ means concatenation. the output has two components: task identification and query answer. we train the unidu model with different multitask learning strategies. dialogue summary and dialogue completion are originally generative tasks. the dialogue contents in the input are the whole dialogue context c and multi-turn utterances h n respectively. since these two tasks are independent of the dialogue domain, there is no domain information in the task query. for dialogue summary, the task query is "what is the summary of this dialogue?". for dialogue completion, the query is ''what is the semantic completion statement of u n ?", where u n is the t-th utterance. their understanding answers are annotated dialogue summaries and rewritten utterances in the output. the original slot filling task demands the model to extract all the mentioned slot values and their slot types in an utterance u n . in this paper, the unidu model predicts the value slot by slot, which is an iterated generation process on the slot candidate list. two different slot filling formats are shown below: i am lakesha mocher b-lname o b-fname o original: unidu: [sf] i am lakesha mocher [c] what is first name on general domain? what is last name on general domain? what is time on general domain? [sf] lakesha [sf] mocher [sf] not mentioned to be clear, we do not list all the candidate slots here. in general, for each sample, it can be formalized as: for dialogue state tracking tasks, the classification methods always achieve better performance than generative methods. however, under the unidu framework, we also formulate dst as a slot-wise value generation task similar to the slot filling task. the dst task formats are shown below: in the dialogue state, its value is "not mentioned", which is a negative sample. note that different utterances are separated by the special token "[t]" in the input. during the training process, the ratio of negative and positive samples is also set below 2:1. for the intent detection task, the original methods formulate it as the intent classification problem and output the distribution of all the candidate intents. the unidu model directly generates the intent name of the current utterance, which can be formalized as: input: [id] u n [c] what is the user's intent on domain d? output: [id] intent name where domain d is normally known in advance. the specific examples of original and unidu formats are shown below: what is the user's intent on the bank business? what can i do if my card still hasn't arrived after 2 weeks? original: unidu: [id] what can i do if my card still hasn't arrived after 2 weeks? [c] [id] card arrival card linking exchange rate card arrival age limit change pin utterance candidate intents intent distribution input output until now, all the five dialogue understanding tasks have been formulated as the unified sequence-tosequence generation task. the specific examples are shown in figure 1 . ## multitask training strategies although the five du tasks can be formulated as a unified generative task, straightforward multitask training may not work due to the different natures of these tasks. in this section, we discuss multitask training strategies and propose a novel model-agnostic adaptive weighting strategy. ## case study we directly validate the unidu model trained with the mats method on unseen "taxi" domain dialogue data collected from multiwoz2.2 corpus. as shown in to further explore the relations among five tasks, we plot the reduced-dimension map of the task embeddings of five tasks with the t-sne algorithm shown in figure 4 . the task embeddings are the final decoder layer representation of the task identification token, whose model is trained with mdts. the dialogue data is from the above unseen "taxi" domain to eliminate the impacts of the dialogue context. we find that the embeddings of dialogue summary, dialogue completion, and intent detection cluster together. these three tasks under the unidu framework are more general than slot filling and dialogue state tracking, whose task queries are slot-wise. the task formats between slot filling and dialogue state tracking are close. however, the unidu model can still have good performance to distinguish between these two tasks. ## related work our work relates to several broad research areas, including prompting, dialogue modeling, and multitask learning. due to the content limitation, here we describe one subarea: multitask learning in nlp applications that relate most closely to our work. @xcite apply a sequence-to-sequence model on three general nlp tasks and study different parameter-sharing strategies. @xcite try to cast nlp tasks as qa over a context. the main topics in this work are how to design an efficient model to integrate the knowledge between question and context. @xcite combine four natural language understanding tasks, which utilize bert as the shared representation model. the model corresponding to each task still has a well-designed part of solving the intrinsic problem. it hampers the analysis of the interaction among the different tasks. recently, @xcite formulated the nlp tasks as the generation task by directly mixing scaling annotated data up. they only focus on zero-shot and few-shot ability on the nlp tasks and ignore the impacts of the different multitask training strategies, which can not achieve better performance on general nlp tasks compared to supervised learning methods on well-designed models. in task-oriented dialogue (tod) modelling, @xcite ; @xcite reformulate the pipeline tod model as the sequential end-to-end generation problem. the end-to-end model needs to generate dialogue state, dialogue action, and response at the same time, which is not scalable when the number of tasks increases. the sequential format needs all the annotations of the same context, which is unavailable in the du area. most recently, pptod @xcite unifies the tod task as multiple generation tasks, including intent detection, dst, and response generation. however, they focus on the response generation ability and ignore the effects of different tasks. in this paper, we deep dive into analyzing the effects of five du tasks. ## conclusion&future work in this paper, we propose a unified generative dialogue understanding framework (unidu) to share the knowledge across five typical dialogue understanding tasks. we introduce a model-agnostic adaptive weight learning method for multitask training to alleviate the biased generation problem. our proposed unidu method achieves better performance compared to well-designed models on a total of five du tasks. we further deep dive into studying the affected factors. finally, experimental results indicate that our proposed unidu model can also get excellent performance under few-shot and zero-shot settings. in the future, we will increase the scale of the du corpora and integrate the unsupervised dialogue pre-training tasks. we will further examine the task-level transferability of the unidu model. mats method. for dialogue state tracking, we utilize the "train" domain data in multiwoz2.2, which is unseen in mtl training phase. compared with vanilla bart, unidu has obvious advantages, especially on extremely resource-limited situation. when there is only 1% and 2% training data, the vanilla bart is disable to learn. unidu model warmed up by mats method can quickly adapt the model on the unseen domain.