Improving End-to-End Task-Oriented Dialog System with A Simple Auxiliary Task (2021)

Abstract: The paradigm of leveraging large pre-trained language models has made significant progress on benchmarks on task-oriented dialogue (TOD) systems. In this paper, we combine this paradigm with multi-task learning framework for end-to-end TOD modeling by adopting span prediction as an auxiliary task. In end-to-end setting, our model achieves new state-of-the-art results with combined scores of 108.3 and 107.5 on MultiWOZ 2.0 and MultiWOZ 2.1, respectively. Furthermore, we demonstrate that multi-task learning improves not only the performance of model but its generalization capability through domain adaptation experiments in the few-shot setting. The code is available at github.com/bepoetree/MTTOD.

## introduction traditional task-oriented dialogue (tod) systems are built on a modular pipeline architecture and their workflow is as follows: the natural language understanding (nlu) module identifies user intents and extracts task-specific slot values, and the dialogue state tracking (dst) module tracks the belief state (i.e., user goal) with considering dialogue history. by using the belief state as a database (db) query, the system can obtain db state, such as the number of matching entities and whether the booking is available. based on the information, the dialogue policy (pol) module determines the next system action and then the natural language generation (nlg) module generates an appropriate natural language response according to the system action. with the advances in neural approach, recent works on tod system handle individual modules in a unified way. in particular, the approach to leveraging the large pre-trained language models for end-to-end dialogue modeling has shown very promising results @xcite @xcite . such models are typically developed by fine-tuning the large pre-trained model, which learned taskagnostic language representations, with only the end-to-end dialogue modeling objective. another approach to leveraging knowledge transfer is multi-task learning, which aims to learn universal representations (knowledge) between related tasks @xcite . it has been shown that multi-task learning not only improves the performance of model, but also mitigates overfitting problem @xcite . furthermore, @xcite demonstrate that the both approaches are complementary and combing them improves the performance of nlu. in this sense, we introduce multi-task learning into fine-tuning an end-to-end tod model, initialized with pre-trained language model. we use t5 @xcite as a backbone and adopt span prediction as an auxiliary task to boost the performance of nlu. our model achieves new state-of-the-art results on both multiwoz 2.0 and multiwoz 2.1 in end-to-end setting. we also investigate the advantages of multi-task learning in end-to-end tod modeling by conducting domain adaptation experiments with the few-shot setting. ## conclusion in this work, we explored the approach to finetuning pre-trained model with multi-task learning for end-to-end tod modeling. our model establishes new state-of-the-art results on both multiwoz 2.0 and multiwoz 2.1 in end-to-end setting. we also demonstrate the effectiveness of multi-task learning in domain adaptation experiments with a few training examples. in future works, we plan to investigate various auxiliary tasks to enhance end-to-end tod modeling.