Evaluating N-best Calibration of Natural Language Understanding for Dialogue Systems (2022)

Abstract: A Natural Language Understanding (NLU) component can be used in a dialogue system to perform intent classification, returning an N-best list of hypotheses with corresponding confidence estimates. We perform an in-depth evaluation of 5 NLUs, focusing on confidence estimation. We measure and visualize calibration for the 10 best hypotheses on model level and rank level, and also measure classification performance. The results indicate a trade-off between calibration and performance. In particular, Rasa (with Sklearn classifier) had the best calibration but the lowest performance scores, while Watson Assistant had the best performance but a poor calibration.

## introduction natural language understanding (nlu) is an important component in dialogue systems. one of the typical tasks of nlu is intent classification: given a user utterance, the nlu returns a list of n hypotheses (an n -best list) ranked according to confidence estimates (a real number between 0 and 1). the highest ranking hypothesis is returned by the nlu as the predicted intent. confidence estimates are also available for lower ranked hypotheses. in this study, we evaluate confidence estimation in 5 nlu services, namely watson assistant, language understanding intelligent service (luis), snips.ai and rasa (with two pipelines @xcite . we measure the calibration and the performance of nlus on rank level (results for a specfic rank) and on model level (aggregated results of all ranks). calibration here refers to the correlation between confidence estimates and accuracies, i.e. how useful the confidence estimate associated with a certain hypothesis is for predicting its accuracy. to achieve our objectives, we conduct an exploratory case study on the 5 nlus. we train the nlus using a subset of a multi-domain dataset proposed by @xcite . we measure the calibration of the nlus on model and rank levels using reliability diagrams and correlation coefficient with respect to instance-level accuracy. we also measure the performance on a model level through accuracy and f1-score. our evaluation aims to facilitate nlu service selection and help dialogue system developers adapt their dialogue system to specific nlu services. for example, depending on the degree of calibration in an nlu, contextual or interactive disambiguation (clarification requests) can be an option. if confidence estimates reflect true accuracy, then if two (or more) hypotheses have similar confidence estimates, this may indicate the presence of an ambiguity in the user input (from the perspective of the nlu, i.e., disregarding dialogue context) that needs to be resolved. conversely, if confidence estimates (especially those for nontop ranks) do not reflect accuracies, then even if the top two (or more) hypotheses have similar estimates, this may not be a reliable indication of ambiguity but rather be due to noise. our evaluation scripts are publicly available on github foot_0 along with the dataset, enabling replication of the study and to ease building on it. ## related work current nlus typically use machine-learning on natural-language data (i.e., the user utterances) to extract features (e.g., keywords, word counts and word embeddings) and predict the intent of the user accordingly @xcite . nlu services are widely used by dialogue developers and allow them to create and train nlu models for dialogue systems. however, the task of choosing the best nlu service depends on the domain and context of the dialogue system. in prior work, benchmarks and evaluations have been performed to identify the best nlu service in different domains like software engineering @xcite , meteorology @xcite , question answering @xcite and others @xcite @xcite . generally, these evaluation studies have been conducted to draw the trade-off line between different nlu services in terms of the usability of their user interfaces @xcite , technical features (e.g., language and device support) @xcite and performance @xcite . nlu performance is usually assessed via performance measures (e.g., accuracy, f1-score, etc.) which depend only on the top hypothesis returned by the nlu, and disregarding the associated confidence estimates. for example, an nlu that predicts 3 out of 10 intents incorrectly with high confidence estimates has the same performance as an nlu that predicts 3 out of 10 intents incorrectly with a low confidence estimation. in earlier work, various methods for visualizing and measuring confidence calibration (the extent to which confidence estimates reflect true likelihoods) have been discussed. for example, @xcite and vasudevan et al. (2019) visualize calibration of neural network models through reliability diagrams. as for quantitative metrics, one proposed measurement is statistical correlation between confidence estimate and some instancelevel performance metric; dong et al. (2018) use spearman's correlation with respect to f1 score, while vasudevan et al. (2019) use pearson's correlation with respect to instance-level accuracy. a second option is to aggregate across instance-level calibration scores (so called proper scoring rules); examples include brier score @xcite and negative log-likelihood @xcite . a third approach involves partitioning confidence estimates into bins, assessing correlation for individual bins, and then aggregating across bin-level calibration results; one popular example of such an approach is expected calibration error (ece) @xcite , which has been extended by @xcite to assess calibration of all predictions rather than only the top one. in this study, we apply some of the previ-ously proposed calibration assessment methodsnamely reliability diagrams and correlation with instance-level performance -to nlus. in addition, we also measure calibration on rank level, enabling a more fine-grained analysis. ## background when using an nlu, an utterance u is fed to the trained nlu, and the output normally includes the information in the following example: the output of the nlu given an utterance u is a prediction consisting of the user utterance, the top intent and an intent ranking. the intent ranking consists of the n -best intent hypotheses along with their corresponding confidence estimates. the confidence estimates reflect how confident the nlu model is regarding each hypothesis. figure 1 illustrates how nlus are used in dialogue systems, involving a scenario where a user asks a dialogue system a question within the home domain. the user utterance (which can be typed by the user in a chat or captured by a speech recognizer) is sent to an nlu service which performs intent classification on the user utterance and returns a prediction with the top intent and the intent ranking. the results are sent to a dialogue manager that decides how to steer the dialogue based on the output from the nlu and some dialogue policy. in case of a high estimated confidence for the most likely hypothesis, the dialogue manager integrates the user's intent, and information is sent to the natural-language generator that generates a response which is uttered back to the user. a dialogue system can use confidence estimates as a basis for choosing a grounding strategy (e.g. asking a control question when confidence is low), ambiguity detection and handling (e.g. asking a clarification question if the top-ranked intents have similar confidence estimates) or re-scoring of hypotheses based on contextual information not available to the nlu but to the dialogue manager (such as dialogue state). different nlus may have different ways of computing confidence estimates, possibly reflecting different notions of confidence. however, for the purpose of using the estimates in a dialogue system, we are interested in how well they reflect true probabilities. in section 4 we note variations in how confidence estimates are computed, but do not take these differences into account in our evaluation. ## nlu services nlu services can be used to construct the nlu component in a dialogue system. in this study, we chose nlu services (henceforth nlus) based on the following criteria: i) can perform intent classification and ii) returns at least 10 top hypotheses in the output. we examine 5 nlus: watson assistant (ibm, 2010), language understanding intelligent service (luis) @xcite , snips (snips, 2013), and rasa @xcite (in two configurations). below, we briefly introduce the nlus. information about the nlus, including the tested version, is summarized in ## dataset and data preparation to conduct intent classification as a part of our evaluation, we build on the dataset proposed by @xcite 25716 user utterances for human-robot interaction and cover 64 intents, 18 scenarios and 21 domains. from this dataset, we select the 10 intents with the most examples (highest number of instances), yielding a total of 14962 utterances (see ## evaluation of confidence estimation an overview of our study's execution is illustrated in figure 2. the evaluation is performed at two lev4 @xcite provide user utterances in different forms: original (raw), with entity annotations, and normalized. in our study, we use the original user utterances. els: rank and model. on rank level, the results are obtained for each rank across the nlus, whereas on model level, the results of all ranks are aggregated. the evaluation focuses on the calibration and performance of the nlus. calibration is measured using reliability diagrams and spearman's correlation coefficient with respect to instancelevel accuracy. the latter is measured through accuracy and f1-score. evaluation is conducted for each split and results are averaged across splits. ## results and analysis in this section we present our results (averaged across the 10 splits). our collected data are visual (reliability diagrams and calibration profiles) and numeric (spearman's correlation, accuracy and f1-score). for our numeric results, we provide the average along with the standard deviation (sd), whereas for the visual results we provide the standard deviation in appendix b to avoid cluttered diagrams. ## discussion in this study, we did not find support for any correlation between calibration and performance (judged by looking only at the top hypothesis). a consequence of this is that when it comes to choosing an nlu for a dialogue system, there is likely to be a trade-off between performance (good for getting the right interpretation) and calibration (good for detecting input that is ambiguous from the nlu perspective). differences in degree of calibration across ranks has been observed for all nlus. specifically, several of the nlus are better calibrated for higherranking hypotheses than for lower-ranking ones. for dialogue system developers, we may interpret this as indicating that it may be useful to look at the top two or three hypotheses when trying to detect ambiguity in input utterances. looking at hypotheses ranked lower than 4 is likely to not be very informative. fortunately, ambiguities are much more frequently 2-way (i.e. there are two possible interpretations of an input) or 3-way than 4-way or more. it is worth stressing that one of the studied nlus (watson) is a multiple-binary classifier (it treats intents independently), while the others are multi-class (they treat intents as mutually exclusive). in this study, we do not investigate whether one type of classifier is more appropriate than another -presumably, both types have benefits and disadvantages. nevertheless, since our dataset assumes a single correct class for a given utterance 7 , our analysis may indirectly favour multi-class classifiers. when interpreting our results, one should also consider that different nlus handle out-of-scope (oos) input differently. specifically, among the 7 utterances in @xcite dataset, on which we build, are labelled with a single correct intent. there are cases of identical utterances for two different intents, but they are very rare (9 out of 25576 unique utterances). ## conclusions and future work we took established calibration measurement approaches and applied them to intent classification of publicly available nlus. we also extended the chosen measurements with a rank-level analysis. our findings show that the best calibrated nlu is rasa-sklearn and the least calibrated nlu is snips, while watson takes the lead as the best performing nlu and rasa-sklearn as the worst performing nlu. the results indicate a trade-off between confidence calibration and performance. we also showed differences in degree of calibration across ranks and discussed their implication (a) accuracy vs. calibration (b) f1-score vs. calibration figure 9: (model-level) accuracy (a) and f1-score (b) vs. calibration for dialogue system development. in future work, it would be interesting to extend the investigation with qualitative analyses of how differences in confidence estimation play out in concrete examples. it could also be valuable to find a better way of assessing how well the nlus capture genuine ambiguity -something which is difficult with a dataset that assumes a single correct intent for a given utterance.