ASSIST : Towards Label Noise-Robust Dialogue State Tracking (2022)

Abstract: The MultiWOZ 2.0 dataset has greatly boosted the research on dialogue state tracking (DST). However, substantial noise has been discovered in its state annotations. Such noise brings about huge challenges for training DST models robustly. Although several refined versions, including MultiWOZ 2.1-2.4, have been published recently, there are still lots of noisy labels, especially in the training set. Besides, it is costly to rectify all the problematic annotations. In this paper, instead of improving the annotation quality further, we propose a general framework, named ASSIST (lAbel noiSe-robuSt dIalogue State Tracking), to train DST models robustly from noisy labels. ASSIST first generates pseudo labels for each sample in the training set by using an auxiliary model trained on a small clean dataset, then puts the generated pseudo labels and vanilla noisy labels together to train the primary model. We show the validity of ASSIST theoretically. Experimental results also demonstrate that ASSIST improves the joint goal accuracy of DST by up to 28.16% on MultiWOZ 2.0 and 8.41% on MultiWOZ 2.4, compared to using only the vanilla noisy labels.

## introduction task-oriented dialogue systems play an important role in helping users accomplish a variety of tasks through verbal interactions @xcite . dialogue state tracking (dst) is an essential component of the dialogue manager in pipeline-based task-oriented dialogue systems. it aims to keep track of users' intentions at each turn of the conversation @xcite . the state information indicates the progress of the conversation and is leveraged to determine the next system action and generate the next system response @xcite . as shown in figure 1 , the dialogue state is typically represented as a set of (slot, value) pairs @xcite . hi, how may i help you? i need to book a room at autumn house. definitely, for how many people and how many nights? just me, 3 nights. can you also give me information on the vue cinema? sure. it is in the city centre, and the phone number is 08451962320. thanks for your help. that's all i need. ## problem definition in this section, we first provide the conventional definition of dst and then extend the definition to the noisy label learning scenario. ## proposed approach we introduce a general framework assist, aiming to train dst models robustly from noisy labels. we assume that a small clean dataset is accessible. based on this dataset, assist first trains an auxiliary model a. then, it leverages a to generate pseudo labels for each sample in the noisy training set. the pseudo state annotations are represented as @xmath0, vt )|@xmath1, where vt denotes the pseudo label of slot s at turn t. afterwards, both the generated pseudo labels and vanilla noisy labels are exploited to train the primary model f * . that is, we intend to learn f * : x @xmath2, bt ), where c( bt , bt ) is a combination of bt and bt . essentially, any existing dst models can be employed as the auxiliary model. however, these models may lead to overfitting due to the small size of the clean dataset. to tackle this issue, we propose a new simple model as the auxiliary model 1 . ## related work in this section, we briefly review related work on dst and noisy label learning. ## conclusion in this work, we have presented a general framework assist, aiming to train dst models robustly from noisy labels. assist leverages an auxiliary model that is trained on a small clean dataset to generate pseudo labels for the large noisy training set. the pseudo labels are combined with the vanilla labels to train the primary model. both theoretical analysis and empirical study have verified the validity of our proposed framework. in the future, we intend to explore more advanced techniques to combine the pseudo labels and vanilla noisy labels in a better way. xiaoxue zang, abhinav rastogi, srinivas sunkara, raghav gupta, jianguo zhang, and jindong chen. 2020. multiwoz 2.2: a dialogue dataset with additional annotation corrections and state tracking baselines. arxiv preprint arxiv:2007.12720. chiyuan zhang, samy bengio, moritz hardt, benjamin recht, and oriol vinyals. 2016. understanding deep learning requires rethinking generalization. arxiv preprint arxiv:1611.03530. jianguo zhang, kazuma hashimoto, chien-sheng wu, yao wang, philip yu, richard socher, and caiming xiong. 2020. find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. in proceedings of the ninth joint conference on lexical and computational semantics, pages 154-167, barcelona, spain (online). association for computational linguistics. zhilu zhang and mert r sabuncu. 2018. generalized cross entropy loss for training deep neural networks with noisy labels. in proceedings of the 32nd international conference on neural information processing systems, pages 8792-8802. qi zhu, kaili huang, zheng zhang, xiaoyan zhu, and minlie huang. 2020. crosswoz: a large-scale chinese cross-domain task-oriented dialogue dataset. transactions of the association for computational linguistics, 8:281-295.