Towards More Accurate Uncertainty Estimation In Text Classification (2020)

Abstract: The uncertainty measurement of classified results is especially important in areas requiring limited human resources for higher accuracy. For instance, data-driven algorithms diagnosing diseases need accurate uncertainty score to decide whether additional but limited quantity of experts are needed for rectification. However, few uncertainty models focus on improving the performance of text classification where human resources are involved. To achieve this, we aim at generating accurate uncertainty score by improving the confidence of winning scores. Thus, a model called MSD, which includes three independent components as “mix-up”, “self-ensembling”, “distinctiveness score”, is proposed to improve the accuracy of uncertainty score by reducing the effect of overconfidence of winning score and considering the impact of different categories of uncertainty simultaneously. MSD can be applied with different Deep Neural Networks. Extensive experiments with ablation setting are conducted on four real-world datasets, on which, competitive results are obtained.

## introduction text classification is a popular topic with broad applications. a successful and common model for text classification is deep neural network (dnn). however, some real-world applications expect results with higher accuracy than the ones achieved by state-of-the-art algorithms. hence, the most uncertain predictions need domain experts for further decisions @xcite . to efficiently leverage the limited human resources, it is essential to calculate uncertainty score of the model prediction, which quantifies how unconfident the model prediction is. this paper aims at generating more accurate uncertainty score through dnns in the text classification with human involvement in the * corresponding author. though various metrics of the uncertainty score have been studied @xcite @xcite @xcite , the existing metrics directly or indirectly depend on winning score, which is the maximum probability in a semantic vector (softmax vector from the last layer of a dnn model) @xcite . therefore, improving confidence of winning score (cws), which describes how confident the winning score matches the sample uncertainty and represents the accuracy of the winning score, is helpful to improve the accuracy of uncertainty score. to show the effect of improving cws, this paper considers a basic way to measure uncertainty score, which is the reciprocal of winning score @xcite . however, we face two challenges in improving cws: (1) how to reduce effect of overconfidence of winning score foot_0 to boost negative correlation between the winning score and sample uncertainty, (2) how to generate winning scores by considering comprehensive categories of uncertainty in one model rather than only one or two categories of uncertainty at a time. the overconfidence of winning scores has been neglected by vast previous works in natural language processing. we identify the presence of overconfidence for the training samples: because the winning scores of training samples are all set as 1 by one-hot labels, each sample will have the same uncertainty score. consequently, the training sample uncertainty will be the same. together, the winning scores and sample uncertainty are the same for various training samples. hence, the negative correlation between the winning scores and sample uncertainty cannot be guaranteed, which is a negative effect of the overconfidence. the effect will affect calculating the uncertainty scores. concretely, in the testing process, we apply different predicted winning scores to match different sample uncertainty based on a latent assumption that the predicted winning score is negatively correlated to the sample uncertainty. however, the assumption is biased because of the negative effect of the overconfidence. to mitigate the impact of overconfidence, we generate new training sample representations with different winning scores, which are also negatively correlated to the sample uncertainty. additionally, the process generating the winning score should consider the impact of different categories of uncertainty simultaneously, while vast of the previous works @xcite @xcite only consider one or two categories of uncertainty at a time 2 . we assume the partial consideration will decrease the cws, and so will the accuracy of uncertainty score. we verify this assumption by our ablation experiments. the uncertainty of a model prediction is derived from two parts: data uncertainty and model uncertainty. the data uncertainty @xcite is further divided into two categories: epistemic uncertainty comes from lack of knowledge, such as only few training data or out-of-distribution testing data; aleatoric uncertainty is caused by noisy data in the generation of both training data and testing data. the model uncertainty @xcite also has two categories: parametric uncertainty comes from different possibilities of parameter values in estimating model parameters under the current model structure and training data; structural uncertainty is uncertainty about whether the current model design (e.g., layers, loss functions) is reasonable or sufficient for the current task and training data. since the solution of structural uncertainty requires extremely high computations, such as neural architecture search (nas) @xcite , we only reduce or scale the other three categories of uncertainty simultaneously to improve the cws. to address the above two challenges, we propose a model called msd, which is named as the initials of its components ("mix-up", "self-ensembling", and "distinctiveness score") aiming at handling overconfidence and various uncertainty with flexibility. the flexibility means that msd is effective on different dnn models (convolutional neural network (cnn), recurrent neural network (rnn) and transformer @xcite ), and each component in msd is independent, which can be 2 please refer to our appendix for detailed comparisons arbitrarily assembled. the main contributions of our work can be summarized as follows, reducing impact of overconfidence. to reduce the impact of overconfidence in calculating uncertainty scores, we apply mix-up to generate new sample representations boosting the negative correlation between the winning scores and sample uncertainty. ## experiments focusing on the text classification with human involvement, we evaluate the performance of msd on four real-world datasets. sec. 4.1 shows an overview of our experiment settings. sec. 4.2 compares the performance between msd and the stateof-the-art methods, and analyzes results of ablation experiments and parameter sensitivity analysis. ## conclusion we aims at generating more accurate uncertainty score to improve the performance of text classification with human involvement. we propose msd with three independent components to improve the cws by mitigating the effect of overconfidence and handling the impact of three categories of uncertainty. msd can be applied to various dnns (cnn, rnn and transformer) and each component in msd can be arbitrarily assembled. extensive experiments on four real-world datasets demonstrate that msd obtains more accurate uncertainty scores, and superiorly improved classification performance when partial uncertain predictions are simulatively assigned to the experts.