Uncertainty Modeling for Machine Comprehension Systems using Efficient B ayesian Neural Networks (2020)

Abstract: While neural approaches have achieved significant improvement in machine comprehension tasks, models often work as a black-box, resulting in lower interpretability, which requires special attention in domains such as healthcare or education. Quantifying uncertainty helps pave the way towards more interpretable neural networks. In classification and regression tasks, Bayesian neural networks have been effective in estimating model uncertainty. However, inference time increases linearly due to the required sampling process in Bayesian neural networks. Thus speed becomes a bottleneck in tasks with high system complexity such as question-answering or dialogue generation. In this work, we propose a hybrid neural architecture to quantify model uncertainty using Bayesian weight approximation but boosts up the inference speed by 80% relative at test time, and apply it for a clinical dialogue comprehension task. The proposed approach is also used to enable active learning so that an updated model can be trained more optimally with new incoming data by selecting samples that are not well-represented in the current training scheme.

## introduction neural approaches demonstrate strong learning capability, achieve significant improvement in various natural language processing tasks @xcite , and are increasingly applied in real-world applications @xcite . however, neural models typically operate as black-box functions and thus lack interpretability. interpreting (or even if only partially) the output from neural models is important in domains such as healthcare, when a model is prone to make incorrect diagnosis @xcite . to tackle this issue, one approach is to evaluate the confidence of an output generated by a model regarding an input, which is through quantifying model uncertainty or epistemic uncertainty. when the uncertainty measure of the model is high, one could be prompted to intervene in the automated decision process by either overriding the system's decision or escalating the situation to a domain expert. this approach would also favor model training with new incoming streams of data that may be ill-represented in the current setting. different from the label probability produced by models, epistemic uncertainty is derived from the weight variance under the observation on a certain distribution. bayesian neural networks (bnns) @xcite , in which prior distributions are applied as additional constraints to weights, have been shown to be effective for quantifying epistemic uncertainty. instead of obtaining deterministic weights, bayesian methods update weights via distribution-based estimation @xcite . therefore, one can sample different possible weights and forward inputs through the network multiple times, then obtain epistemic uncertainty according to the variance of a set of predictions. moreover, drawing experience from past work, modeling within a bayesian framework can lead to potentially better representations and predictions in various tasks @xcite . most previous studies applied bayesian neural approaches on classification or regression tasks. in this paper, we focus on modeling and utilizing epistemic uncertainty for question-answering (qa) systems in natural language processing, and tackling the aforementioned issues in the healthcare domain. since the neural network architectures for qa tasks are relatively more complex, there is a need to balance the learning quality and inference speed. to this end, we propose a hybrid neural architecture by integrating bayesian approximation to a base neural model and optimize its training strategy. we conduct experiments on a clinical conversational scenario in section 4.1 and a question-answering benchmark dataset (see appendix b). the result shows that our approach can achieve better performance and is capable of modeling epistemic uncertainty. furthermore, we analyze the characteristics of the quantified uncertainties and conduct an active learning experiment on the clinical corpus. ## in relation to other work neural question-answering approaches, often applied to machine comprehension tasks, have achieved rapid progress lately, benefiting from large-scale corpora @xcite , semantic vector representations @xcite , sophisticated neural architectures @xcite , and deep contextual language models @xcite , pushing the state-of-the-art performance on various benchmarks. however, the extent to which these systems truly understand language remains unclear, and models are vulnerable to adversarial samples @xcite . as an effective approach to model weight variance and generate predictions, bayesian neural networks and their variants have been applied in computer vision for image classification @xcite and autonomous vehicles to better model safety @xcite . in natural language processing, uncertainty modeling has been adopted in sentiment analysis, named entity recognition and language modeling @xcite . such approaches have also proved effective in domain-specific active learning such as named entity recognition @xcite . to the best of our knowledge, we take the first stab to introduce neural epistemic uncertainty modeling in question-answering tasks. making bayesian neural networks tractable on large-scale practical problems has been a focus in the research field since the 1990's @xcite @xcite . more recently, several approximation methods have been proposed, including bayes-by-backprop @xcite , which places a prior distribution over model parameters and calculates the kullback-leibler (kl) divergence between approximated and expected posterior distribution; and monte-carlo dropout @xcite , which applies dropout in both training and inference stages to approximate bayesian variational inference. sampling from approximated posterior distribution using gradient uncertainty can also be used to represent uncertainty in predictions @xcite . ## conclusion in this work, we defined how to quantify epistemic uncertainty in question-answering tasks. we further proposed a hybrid neural architecture that achieves performance comparable to regular bayesian neural networks but offers greater efficiency, speeding up the inference processing time by 80% relative. the proposed approach also enabled active learning for dialogue comprehension tasks so that an updated model was trained more optimally with new incoming data by selecting training samples that may not have been well-represented in the current training dataset.