Towards Zero and Few-shot Knowledge-seeking Turn Detection in Task-orientated Dialogue Systems (2021)

Abstract: Most prior work on task-oriented dialogue systems is restricted to supporting domain APIs. However, users may have requests that are out of the scope of these APIs. This work focuses on identifying such user requests. Existing methods for this task mainly rely on fine-tuning pre-trained models on large annotated data. We propose a novel method, REDE, based on adaptive representation learning and density estimation. REDE can be applied to zero-shot cases, and quickly learns a high-performing detector with only a few shots by updating less than 3K parameters. We demonstrate REDE’s competitive performance on DSTC9 data and our newly collected test set.

## introduction current task-oriented dialog systems often rely on pre-defined apis to complete target tasks @xcite and filter out any other requests beyond the apis as out-of-domain cases. however, some in-domain user requests can be addressed by incorporating external domain knowledge from the web or any other sources @xcite . to address this problem, @xcite recently organized a benchmark challenge on taskoriented conversational modeling with unstructured knowledge access in dstc9 @xcite . this challenge includes the knowledgeseeking turn detection task to determine whether to invoke a knowledge-driven responder or just rely on available api functions. one data sample is provided in to evaluate the detector performance on real-world user queries, we specially curate a new contrast set following @xcite by manually collecting questions posted by real users on tripadvisor forums. we found that the detector trained on dstc9 track 1 training samples had a large performance degradation on this contrast set (f1 score dropped by over 15%), suggesting the need for methods with better generalization. in this work, we propose a method that can quickly learn a knowledge-seeking turn detector with much fewer out-of-domain samples, such as only a few shots or even zero shot. our method is composed of two stages: representation learning and density estimation (rede). first, we learn a representation model via fine-tuning a pre-trained sentence encoder on all non-knowledge-seeking turns (utterances that can be supported by apis) via masked language modeling (mlm). then we learn a density estimator using these representation vectors. during inference, the density estimator produces a density score for a given user utterance. if it is above a threshold, this utterance is counted as an in-domain api turn, otherwise as a knowledge-seeking turn. to incorporate out-of-domain examples, we propose to use principle component analysis to quickly learn a projection matrix with few knowledgeseeking turn samples and then use this matrix to linearly transform the representation vectors. we conduct experiments on the dstc9 track 1 data as well as our new contrast test set. we demonstrate that rede can achieve competitive performance as other supervised methods in the full-shot setting and outperform them by a large margin in the lowresource setting. more importantly, our approach generalizes much better in the new contrast test set that we created. overall, our contributions are summarized as follows: our work is closely related to those participating systems in dstc9 track 1 @xcite . all the systems proposed to treat the problem of knowledge-seeking turn detection as a binary classification task and fine-tuned pre-trained models such as roberta, unilm, plato, gpt2, on the whole training set @xcite @xcite , which yielded around 99% and 96% f1 scores on the development and test sets, respectively. our method differs in two aspects: 1) we do not need to fine-tune the pre-trained model on the training set; 2) our model is at least 5 times smaller and we need less than 5% of training data to achieve similar performance. our method is inspired by previous work for out-of-domain (ood) detection @xcite @xcite and one-class classification @xcite . @xcite also tried tackling this problem by applying an unsupervised anomaly detection algorithm, local outlier factor (lof) @xcite , which compares the local densities between a given input instance and its nearest neighbors, but did not obtain good results (f1 score is less than 50%). @xcite proposed to first learn a representation model via contrastive learning, then learn a density estimator on the obtained representations. they showed decent performance for one-class classification. all these previous work assumed no access to ood samples, however, we would like to make use of those ood samples efficiently when they are available. therefore we extend the general representation learning framework by proposing a novel representation transformation method to learn ood samples, which leads to significantly boosted detection performance. our work is also related to few/zero-shot learn-ing, which has been widely studied previously @xcite . transfer learning @xcite @xcite and data augmentation @xcite have been two major methods for this direction, while our work focuses on manipulation of learned representations, which provides a new perspective. ## methods our method includes three steps: encoder adaptation, representation transformation, and density estimation. the representation transformation step is only applicable when there are ood examples (i.e., knowledge-seeking turns). ## conclusion in this work, we propose a novel method rede based on domain-adapted representation learning and density estimation for knowledge-seeking turn detection in tasked-orientated dialogue systems. compared with previous sota models, rede can achieve comparable performance in the full supervised setting and significantly superior performance for the low-resource setting. besides, rede has much better generalization capability onto a new contrast set we curated.