D ialog VCS : Robust Natural Language Understanding in Dialogue System Upgrade (2024)

Abstract: In the constant updates of the product dialogue systems, we need to retrain the natural language understanding (NLU) model as new data from the real users would be merged into the existing data accumulated in the last updates. Within the newly added data, new intents would emerge and might have semantic entanglement with the existing intents, e.g. new intents that are semantically too specific or generic are actually a subset or superset of some existing intents in the semantic space, thus impairing the robustness of the NLU model.As the first attempt to solve this problem, we setup a new benchmark consisting of 4 Dialogue Version Control dataSets (DialogVCS). We formulate the intent detection with imperfect data in the system update as a multi-label classification task with positive but unlabeled intents, which asks the models to recognize all the proper intents, including the ones with semantic entanglement, in the inference.We also propose comprehensive baseline models and conduct in-depth analyses for the benchmark, showing that the semantically entangled intents can be effectively recognized with an automatic workflow. Our code and dataset are available at https://github.com/Zefan-Cai/DialogVCS .

## introduction with the rapid growth of the business market for the task-oriented chatbots, the service providers would constantly upgrade their dialogue systems in order to be adaptable to the changing user requirements. within the system update, the workflow of updating the existing natural language understanding (nlu) model is to collect a new training corpus by accumulating emerging data and merging them into the existing training data in the last iteration, followed by retraining with the updated corpus. throughout play_music play_video play_media book_train _ticket book_ticket book_hotel book_trip play_music play_video book_tickets book_hotels play_media book_trip play_music_on_repeat book_train_tickets book_flight_tickets intents system updates update v n update v m ... ... play_music_ on_repeat book_flight _ticket add corpus&intents add corpus&intents ... the model update, new intents would emerge as more and more real-world user queries arrive. the prior research on nlu focused on the utterance understanding with a well-defined intent 1 ontology, with the assumption that the entire intents are semantically separable and organized in the proper granularity 2 . however, the emerging intents from nlu model update might be incompatible with the existing intent ontology and thus violate the assumption regarding the properties of being semantically non-overlapping and maintain-ing well-designed granularity, e.g. the emerging intents 'play_music_on_repeat' and 'play_media' are semantically too specific or generic with respect to the existing intent 'play_music'. we categorize the semantic overlapping problem between the emerging and the existing intents among the system upgrade into two categories, namely version conflict and merge friction, in which the version conflicts signify the emerging intents are too semantically specific and thus covered by the existing intents while the merge frictions are just the opposite. we argue that the semantic overlapping problem between emerging and existing intents occurs frequently in the dialogue system updates as the careful human modification for each emerging intent would be prohibitive due to the limited labor budgets and the imminent product delivery deadlines. the defective data would even propagate and accumulate through consecutive upgrades, and thus largely impair the robustness of the nlu models. we formulate the problem as a multi-label classification task with positive but unlabeled intents foot_1 . as the first step towards solving this problem, we setup a benchmark consisting of 4 dialogue version control datasets (dialogvcs) to simulate the semantically overlapped intents. we employ a fully automatic workflow to create the atis-vcs, snips-vcs, multiwoz-vcs, crosswoz-vcs datasets from 4 canonical nlu datasets, including atis @xcite , snips @xcite , multiwoz @xcite and crosswoz @xcite , by splitting the original intents according to the pivot entities or intentions. by leveraging existing high-quality datasets, it provides a distinct advantage in terms of quality assurance. on the other hand, manual annotation on real scenario data could be challenging to maintain consistent quality. the most critical challenge of dialogvcs is the discrepancy between training and inference, i.e. for each training instance, only one intent is provided as the target label foot_2 , while in the testing phase, the models are expected to output all the ground-truth labels. thus we setup multiple baselines concerning with positive but unlabeled (pu) learning for the proposed benchmark and find that the baseline models are capable of detecting semantically overlapped intents automatically. we summarize our contributions below: 1) we model the version conflicts and merge frictions of nlu models in the industrial dialogue system update as a multi-label classification task with positive but unlabeled intents, making it accessible to the research community. 2) we propose 4 dialogue version control datasets by simulating the semantic overlapping problem on the atis, snips, multiwoz, and crosswoz datasets. 3) we setup various baselines for the proposed benchmark and show that the semantically overlapping intents can be effectively detected with an automatic workflow. formulations to better signify the two aforementioned challenges, suppose at first we have two intents i 1 and i 2 , the version conflict would occur when the new intents i v1 1 , i v2 1 emerges where the superscripts v1 and v2 imply that i v1 1 and i v2 1 are different labels with respect to i 1 but semantically identical; the merge friction would occur as the new intent i 1 &i 2 appears where the ampersand emphasizes the new intent is different but semantically affiliated to i 1 and i 2 . note that i 1 , i v1 1 and i 1 &i 2 are just the notations of the given intents rather than the real intent names, which means we can not know the relations among these intents a priori. ## methods we highlight the technical challenges of di-alogvcs: 1) the discrepancy between training and testing due to the positive but unlabeled (pu) setting; 2) the risk of pivoting the model training with false negative labels; 3) the extreme 0-1 class imbalance of multi-label classification. we propose multiple baselines towards these challenges. ## related work robust nlu recently, the topics concerning the nlp robustness and debiasing have attracted board attention @xcite . to the best of our knowledge, this study is the first to investigate the non-robustness of nlu systems caused by overlapping and conflicting labels resulting from continuous system updates. ## conclusion the version conflicts and merge frictions of intents occur frequently due to the semantic overlapping between emerging and existing intents in the industrial dialogue system updates, but are unexplored in the research community. we take the first step to model the version conflict problem as a multilabel classification with positive but unlabeled intents, and propose a dialogue version control (di-alogvcs) benchmark with extensive baselines. we find that the overlapping intents can be effectively detected with an automatic workflow. we leave the construction of real-scenario data for future works.