Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance (2021)

Abstract: The ability to identify and resolve uncertainty is crucial for the robustness of a dialogue system. Indeed, this has been confirmed empirically on systems that utilise Bayesian approaches to dialogue belief tracking. However, such systems consider only confidence estimates and have difficulty scaling to more complex settings. Neural dialogue systems, on the other hand, rarely take uncertainties into account. They are therefore overconfident in their decisions and less robust. Moreover, the performance of the tracking task is often evaluated in isolation, without consideration of its effect on the downstream policy optimisation. We propose the use of different uncertainty measures in neural belief tracking. The effects of these measures on the downstream task of policy optimisation are evaluated by adding selected measures of uncertainty to the feature space of the policy and training policies through interaction with a user simulator. Both human and simulated user results show that incorporating these measures leads to improvements both of the performance and of the robustness of the downstream dialogue policy. This highlights the importance of developing neural dialogue belief trackers that take uncertainty into account.

## introduction in task-oriented dialogue, the system aims to assist the user in obtaining information. this is achieved through a series of interactions between the user and the system. as the conversation progresses, it is the role of the dialogue state tracking module to track the state of the conversation. for example, in a restaurant recommendation system, the state would include the information about the cuisine of the desired restaurant, its area as well as the price range that the user has in mind. it is crucial that this state contains all information necessary for the dialogue policy to make an informed decision for the next action @xcite . policy training optimises decision making in order to complete dialogues successfully. it has been proposed within the partially observable markov decision process (pomdp) approach to dialogue modelling to track the distribution over all possible dialogue states, the belief state, instead of a single most-likely candidate. this approach successfully integrates uncertainty to achieve robustness @xcite @xcite . however, such systems do not scale well to complex multidomain dialogues. on the other hand, discriminative neural approaches to dialogue tracking achieve state-of-the-art performance in the state tracking task. nevertheless, the state-of-the-art goal accuracy on the popular multiwoz @xcite multi-domain benchmark is currently only at 60% @xcite . in other words, even the best neural dialogue state trackers at present incorrectly predict the state of the conversation in 40% of the turns. what is particularly problematic is that these models are fully confident about their incorrect predictions. unlike neural dialogue state trackers, which predict a single best dialogue state, neural belief trackers produce a belief state @xcite . state-of-the-art neural belief trackers, however, achieve an even lower goal accuracy of approximately 50% @xcite , making the more accurate state trackers a preferred approach. high-performing state trackers typically rely on span-prediction approaches, which are unable to produce a distribution over all possible states as they extract information directly from the dialogue. ensembles of models are known to yield improved predictive performance as well as a calibrated and rich set of uncertainty estimates @xcite . unfortunately, ensemble generation and, especially, inference come at a high computational and memory cost which may be pro-hibitive. while standard ensemble distillation @xcite can be used to compress an ensemble into a single model, information about ensemble diversity, and therefore several uncertainty measures, is lost. recently @xcite and @xcite proposed ensemble distribution distillation (end 2 ) -an approach to distill an ensemble into a single model which preserves both the ensemble's improved performance and full set of uncertainty measures at low inference cost. in this work we use end 2 to distill an ensemble of neural belief trackers into a single model and incorporate additional uncertainty measures, namely confidence scores, total uncertainty (entropy) and knowledge uncertainty (mutual information), into the belief state of the neural dialogue system. this yields an uncertainty-aware neural belief tracker and allows downstream dialogue policy models to use this information to resolve confusion. to our knowledge, ensemble distillation, especially ensemble distribution distillation, and the derived uncertainty estimates, have not been examined for belief state estimation or any downstream tasks. we make the following contributions: 1. we present setsumbt, a modified sumbt belief tracking model, which incorporates set similarity for accurate state predictions and produces components essential for policy optimisation. 2. we deploy ensemble distribution distillation to obtain well-calibrated, rich estimates of uncertainty in the dialogue belief tracker. the resulting model produces state-of-the-art results in terms of calibration measures. 3. we demonstrate the effect of adding additional uncertainty measures in the belief state on the downstream dialogue policy models and confirm the effectiveness of these measures both in a simulated environment and in a human trial. ## effects of uncertainty on downstream tasks we take the following steps in order to examine the effects of the additional uncertainty measures in the dialogue belief state: 1. modify the original sumbt model @xcite to arrive at a competitive baseline. we call this model setsumbt. 2. produce ensembles of setsumbt following the work of van niekerk et al. (2020). 3. apply end and end 2 as introduced in section 2.3. 4. apply policy optimisation that uses belief states from distilled models. ## conclusion whilst neural dialogue state trackers may achieve state-of-the-art performance in the isolated dialogue state tracking task, the absence of uncertainty estimates may lead to less robust performance of the downstream dialogue policy. in this work we propose the use of total and knowledge uncertainties along with confidence scores to form a dialogue belief state. we moreover describe a model, set-sumbt, that can produce such a belief state via distillation. experiments with both simulated and real users confirm that these uncertainty metrics can lead to more robust dialogue policy models. in future, we will investigate modifying span-based dialogue state trackers to incorporate uncertainty. we will further investigate the expansion of the set-sumbt model to include the correlation between different domain-slot pairings.