“None of the Above”: Measure Uncertainty in Dialog Response Retrieval (2020)

Abstract: This paper discusses the importance of uncovering uncertainty in end-to-end dialog tasks and presents our experimental results on uncertainty classification on the processed Ubuntu Dialog Corpus. We show that instead of retraining models for this specific purpose, we can capture the original retrieval model’s underlying confidence concerning the best prediction using trivial additional computation.

## introduction uncertainty modeling is a widely explored problem in dialog research. stochastic models like deep qnetworks @xcite , gaussian processes @xcite , and partially observable markov decision process @xcite are often used in spoken dialog systems to optimize dialog management by explicitly estimating uncertainty in policy assignments. however, these approaches are either computationally intensive @xcite or require significant work on refining policy representations @xcite . moreover, most current uncertainty studies in dialog focus on the dialog management component. end-to-end (e2e) dialog retrieval models jointly encode a dialog and a candidate response @xcite , assuming the ground truth is always present in the candidate set, which is not the case in production. @xcite recently showed that classifiers that perform well on in-scope intent classification for task-oriented dialog systems struggle to identify out-of-scope queries. the response selection task in the most recent dialog system technology challenge (chulaka gunasekara and lasecki, 2019) also explicitly mentions that "none of the proposed utterances is a good candidate" should be a valid option. the goal of this paper is to set a new direction for future task-oriented dialog system research: while retrieving the best candidate is crucial, it should be equally important to identify when the correct response (i.e. ground truth) is not present in the candidate set. in this paper, we measure the e2e retrieval model's capability to capture uncertainty by inserting an additional "none of the above" (nota) candidate into the proposed response set at inference time. the contributions of this paper include: (1) demonstrating that it is crucial to learn the relationship amongst the candidates as a set instead of looking at point-wise matching to solve the nota detection task. as a result, the logistic regression (logreg) approach proposed here consistently achieves the best performance compared to several strong baselines. (2) extensive experiments show that the raw output score (logits) is more informative in terms of representing model confidence than normalized probabilities after the softmax layer. ## related work our use of nota to measure uncertainty in dialog response is motivated by the design of student performance assessment in psychology studies. test creators often include nota candidates in multiple-choice design questions, both as correct answers and as distractors. how the use of nota affects the difficulty and discrimination of a question has been discussed widely @xcite . for assessment purposes, a common finding is that using nota as the correct response increases question difficulty, and also lures high-and low-performing students toward distractors @xcite . returning a nota-like response is a common practice in dialog production systems (ibm). the idea of adding the nota option to a candidate set is also widely used in other language technology fields like speaker verification @xcite . however, the effect of adding nota is rarely introduced in dialog retrieval research problems. to the best of our knowledge, we are the first to scientifically evaluate a variety of conventional approaches for retrieving nota in the dialog field. ## results and analysis figure 4 shows the average f1 score trends with the original selection model on the test set with 100 distractors. the plot shows the trend that with more distractors, the lstm model struggles to determine the presence of ground truth, while the logreg model performs consistently well. the complete results of this extended test set are in the appendix. ## discussion with nota options in the training data, the models learn to sometimes predict nota as the best response, resulting in more false-positive isnota predictions at inference time. also, by replacing various ground truths and strong distractors with nota, the model has fewer samples to help it learn to distinguish between different ground truths and strong distractors/ thus it performs less well on borderline predictions (scores close to the threshold). this behavior results in some selection methods trained on the dataset containing nota tokens performing worse than when they are trained on the original dataset. this motivates us to advocate the proposed logreg approach instead of the conventional add a nota choice method. another prominent advantage of the logreg approach is that it does not require data-or modeldependent input like embedding vectors or hidden layer output. instead, it takes logits or normalized scores, both of which can be output from any models. this feature makes our approach insensitive to the underlying architecture. ## conclusions we have created a new nota task on the ubuntu dialog corpus, and have proposed to solve the problem by learning the response set representation with a binary classification model. we hope the dataset we release will be used to benchmark future dialog system uncertainty research.