GOLD : Improving Out-of-Scope Detection in Dialogues using Data Augmentation (2021)

Abstract: Practical dialogue systems require robust methods of detecting out-of-scope (OOS) utterances to avoid conversational breakdowns and related failure modes. Directly training a model with labeled OOS examples yields reasonable performance, but obtaining such data is a resource-intensive process. To tackle this limited-data problem, previous methods focus on better modeling the distribution of in-scope (INS) examples. We introduce GOLD as an orthogonal technique that augments existing data to train better OOS detectors operating in low-data regimes. GOLD generates pseudo-labeled candidates using samples from an auxiliary dataset and keeps only the most beneficial candidates for training through a novel filtering mechanism. In experiments across three target benchmarks, the top GOLD model outperforms all existing methods on all key metrics, achieving relative gains of 52.4%, 48.9% and 50.3% against median baseline performance. We also analyze the unique properties of OOS data to identify key factors for optimally applying our proposed method.

## introduction detecting out-of-scope scenarios is an essential skill of dialogue systems deployed into the real world. while an ideal system would behave appropriately in all conversational settings, such perfection is not possible given that training data is finite, while user inputs are not @xcite . out-of-distribution issues occur when the model encounters situations not covered during training, including novel user intents, domain shifts or custom entities @xcite . unique to conversations, dialogue breakdowns represent cases where the user cannot continue the interaction with the system, perhaps due to ambiguous requests or prior misunderstandings @xcite . such breakdowns might fall within the distribution of plausible utterances, yet still fail to make sense due to the given context. oos detection aims to recognize both out-of-distribution problems and dialogue breakdowns. prior methods tackling oos detection in text have shown great promise, but typically assume access to a sufficient amount of labeled oos data during training @xcite , which is unrealistic in open-world settings @xcite . alternative methods have also been explored which train a supporting model using in-scope data rather than directly training a core model to detect oos instances @xcite . as a result, they suffer from a mismatch where the objective during training does not line up with the eventual inference task, likely leading to suboptimal performance. more recently, data augmentation techniques have been applied to in-scope (ins) data to improve out-of-domain robustness @xcite . however, we hypothesize that since ins data comes from a different distribution as oos data, augmentation on the former will not perform as well as augmentation on the latter. in this paper, we propose a method of generating out-of-scope labels with data augmentation (gold) to improve oos detection in dialogue. to create new pseudo-labeled examples, we start with a small seed set of known oos examples. next, we find utterances that are similar to the known oos examples within an auxiliary dataset. we then generate candidate labels by replacing text from the known oos examples with the similar utterances uncovered in the previous step. lastly, we run an election to filter down the candidates to only those which are most likely to be out-of-scope. our method is complementary to other indirect prediction techniques and in fact takes advantage of progress by other methods. we demonstrate the effectiveness of gold across three task-oriented dialogue datasets, where our method achieves state-of-theart performance across all key metrics. we conduct extensive ablations and additional experiments to probe the robustness of our best performing model. finally, we provide analysis and insights on augmenting oos data for other dialogue systems. ## background and baselines in this section we formally describe the task of outof-scope detection and the different approaches to handling this issue. ## gold to avoid a mismatch between training and inference, we are motivated to explore the direct prediction paradigm in a way the does not violate the oos data restriction inherent to indirect prediction methods. concretely, gold performs data augmentation on a small sample of labeled oos examples to generate pseudo-oos data. this weaklylabeled data is then combined with ins data for training a core oos detector. we limit the number of oos samples to be only 1% of the size of inscope training examples. note that indirect methods also typically have access to a modest number of oos samples for tuning hyper-parameters, such as thresholds, so this adjustment is not an exclusive advantage of our method. in addition to a small seed set of oos examples, we assume access to an external pool of utterances, which serve as the source of data augmentations, similar to @xcite . we refer to this auxiliary data as the source dataset s, as opposed to the target dataset t used for evaluating our method. gold now proceeds in three basic steps. (see algorithm 1 for full details.) ## key results we now present the results of our main experiments. as evidenced by figure 3 , mix performed as the best data source across all datasets, so we use it to report our main metrics within ## discussion and analysis in this section, we conduct follow-up experiments to analyze the impact of our method's components and identify best practices when applying data augmentation for oos detection. ## conclusion this paper presents gold, a method for improving oos detection when limited training examples are available by leveraging data augmentation. rather than relying on a separate model to support the detection task, our proposed method directly trains a model to detect out-of-scope instances. compared to other data augmentation methods, gold takes advantage of auxiliary data to expand the coverage of out-of-scope distribution examples rather than trying to extrapolate from in-scope examples. moreover, our analysis reveals key techniques for further diversifying the training data to support robustness and prevent overfitting. we demonstrate the effectiveness of our technique across three dialogue datasets, where our top models outperform all baselines by a large margin. future work could explore detecting more granular levels of errors, as well as more sophisticated methods of filtering candidates @xcite .