Uncertainty Estimation and Reduction of Pre-trained Models for Text Regression (2022)

Abstract: State-of-the-art classification and regression models are often not well calibrated, and cannot reliably provide uncertainty estimates, limiting their utility in safety-critical applications such as clinical decision-making. While recent work has focused on calibration of classifiers, there is almost no work in NLP on calibration in a regression setting. In this paper, we quantify the calibration of pre- trained language models for text regression, both intrinsically and extrinsically. We further apply uncertainty estimates to augment training data in low-resource domains. Our experiments on three regression tasks in both self-training and active-learning settings show that uncertainty estimation can be used to increase overall performance and enhance model generalization.

## introduction modern neural network models, particularly those based on pre-training and fine-tuning, have achieved impressive results across a broad spectrum of nlp tasks, in terms of evaluation metrics such as classification accuracy or f-score for classification tasks and mean squared error for regression tasks. however, the standard training regime fails to take model uncertainty into account, and tends to result in over-fitting and poor generalization, especially in limited training data situations. in addition, these models have been empirically demonstrated to have poor calibration-the predictive probability does not reflect the true correctness likelihood, and they are generally overconfident when they make wrong predictions @xcite @xcite . put differently, the models do not know what they don't know. this is particularly the case in low-resource settings. however, faithfully assessing the uncertainty of model predictions is as important as obtaining high accuracy in many safety-critical applications, such as autonomous driving or clinical decision support @xcite @xcite . if models were able to more faithfully capture their lack of certainty when they make erroneous predictions, they could be used more reliably in critical decision-making contexts, and avoid catastrophic errors. in the context of text regression, we aim to alleviate over-fitting and improve generalizability in low-resource settings by taking the uncertainty sourced from both the data and model into account. specifically, we address: (1) data uncertainty by filtering noisy annotations from (either pseudo or gold) labeled data based on predictive confidence, preventing models from memorizing out-of-distribution examples; and (2) model uncertainty to accurately estimate both the target value and predictive confidence by uncertainty models, providing more reliable and interpretable predictions, meanwhile effectively supporting denoising in (1). uncertainty estimation has been extensively explored in the context of classification @xcite @xcite , but is relatively unexplored for regression tasks, due to the complexities in dealing with a continuous target space. the output of a classifier passed through a softmax layer naturally provides a discrete probability distribution, while in a regression setting the output is a single numerical value. we compare four well-studied techniques for uncertainty estimation, as applied to pre-trained language models (lms): gaussian processes @xcite , bayesian linear regression (hernández-lobato and adams, 2015), bayes by backprop, and monte carlo (mc) dropout. to comprehensively assess uncertainty quality, we evaluate results intrinsically using various metrics, and extrinsically with several downstream experiments. our analysis shows that predictions are highly uncertain and inaccurate in low-resource scenarios. two major types of uncertainty have been identified: aleatoric uncertainty captures noise inherent in the observations; and epistemic uncertainty accounts for uncertainty in the model, which can be explained away given enough data, compensating for limited knowledge @xcite . in other words, uncertainty results primarily from noisy human annotations, insufficient labeled data, and out-of-domain text in practice @xcite . we therefore propose a simple method to filter noisy labels and select high-quality instances from an unlabeled data pool based on the predictive confidence, which on the one hand alleviates both aleatoric and epistemic uncertainty, and on the other hand, improves accuracy and generalization thanks to increased training data. in this work, we explore how to estimate uncertainty in a regression setting with pre-trained language models, and evaluate estimation quality both intrinsically and extrinsically. intrinsic uncertainty estimation provides the basis for our proposed data selection strategy: by filtering noise based on confidence thresholding, and mitigating exposure bias, our approach is shown to be effective at improving both performance and generalization in low-resource settings, in self-training, and active learning settings. ## background we first review approaches for estimating the predictive uncertainty of deep neural networks (dnns) in a regression setting, then methods for reducing uncertainty and improving generalization. ## tasks and notation in this paper, we consider text regression across three separate tasks, and a total of 10 datasets. tasks sts: semantic textual similarity assesses the degree of semantic equivalence between two pieces of text @xcite . the aim is to predict a similarity score for a sentence pair (s1, s2), generally in the range [0, 5], where 0 indicates complete dissimilarity and 5 indicates equivalence in meaning. as an example: s1: total minutes spent in timed codes: 10 mins. s2: total minutes spent in timed codes: 33 mins. might be labeled 4, as the two texts differ only in very specific content (underlined). sa: sentiment analysis rating involves predicting a sentiment score for a review s, in the range 1 (extremely negative) to 5 (extremely positive). da: machine translation quality estimation, based on the direct assessment approach @xcite , aims to predict a normalised quality score for text pair (s1, s2), where s2 is machine translated from s1. as such, it is similar to sts, but differs in that it is cross-lingual. notation and assumptions throughout this paper, raw examples, column vectors, and matrices are denoted in lower-case italics, bold, and upper-case italics, respectively (e.g., x, x, and x). θ encoder and θ reg represent parameters of the encoder and task-specific regression layers, and f (θ, •) refers to the whole model. take a dataset @xmath0, y 1 ), ( x loss function is the empirical risk of the mean square error (mse): @xmath1 datasets we evaluate on different-sized datasets across various domains for sts and sa, and three same-sized datasets for da, summarized in for sts, we use: (1) one large-scale general dataset, sts-b @xcite ; (2) two small clinical data sets, medsts @xcite and n2c2-sts @xcite ; and (3) two small biomedical data sets, biosses @xcite and ebmsass @xcite , each of which is 5-way annotated. for sa, we use: (1) a large-scale product review dataset, yelp @xcite ; and (2) a small paper review rating dataset, peerread @xcite , augmented with 399 spanish paper reviews @xcite machine-translated into english. for da, we use the three language pairs from wmt2020 @xcite , en-zh, ru-en, and si-en, corresponding to high-, medium-, and lowresource settings in terms of the source language. ## method in this section, we first introduce approaches for estimating regression uncertainty based on pre-trained lms, then propose a simple method to sample ''clean'' instances from unlabeled data to augment training data based on predictive uncertainty. the proposed methods can be applied equally in semi-supervised and unsupervised settings (including active learning and self-learning). ## uncertainty evaluation metrics evaluating uncertainty estimates of predictions is challenging in a regression setting, as the ''ground truth'' uncertainty is usually not available @xcite . to evaluate model predictions, we consider four metrics. pearson correlation: it is vital to assess the predictive accuracy of the system, regardless of the uncertainty estimate. we use pearson correlation r to evaluate the correlation between the system's average predictions and ground truth quality scores. calibration error (cal): one way to understand if models can be trusted is by analysing whether they are calibrated. @xcite defined calibration in a regression setting as the asymptotic consistency between the probabilistic forecasts f i and the true data-generating distributions g i , with the index i referring to each example. practically, f i is the cumulative probability distribution p (@xmath2, g i is generally estimated by empirical distribution functions based on the observations only. so calibration measures if the predictive confidence estimates are aligned with the empirical correctness likelihoods. given a confidence level p j , the empirical accuracy is calculated: negative log-probability density (nlpd) complements cal's equal treatment to over-and under-confidence. it penalises over-confidence more strongly through logarithmic scaling: sharpness (shp): the metrics above do not account for the concentration of the predictive distributions, which generally favours predictors that produce wide and uninformative confidence intervals. to guarantee useful uncertainty estimation, confidence intervals should not only be calibrated, but also sharp and ''tight'' around the predicted value. the numerical width of prediction intervals @xcite and the mean of variance @xcite are often used to quantify sharpness. we apply the latter in our work, with a lower score implying higher sharpness. to interpret mixed results, for example when a model attains the best sharpness but with infinitely large nlpd, we suggest that pearson correlation (r) has primacy, followed by cal and nlpd, then shp. that is, when models have comparable r, the comparison of cal/nlpd is more meaningful, and if those are also similar, shp should be considered; otherwise, it's largely meaningless. ## evaluation of uncertainty estimation we expect that the incorporation of uncertainty estimation should not harm predictive performance compared to point estimation without uncertainty, in both in-and out-of-domain scenarios. additionally, uncertainty estimates should reflect ''what the model does not know'', making it possible to determine whether a prediction can be trusted based on the output distribution. this is quantified intrinsically with cal and nlpd (the lower, the better), and extrinsically via instance selection in section 7. ## analysis in this section, we conduct further analysis to better understand the results of the experiments. qualitative comparison: in both in-domain and out-of-domain evaluation, end-to-end training based on bert, particularly bbb estimation, obtains much larger nlpd than pipeline training based on sbert, especially gp regression. we speculate that end-to-end uncertainty models are confident for both correct and incorrect we score sentence pairs in the sts-b test set using bert bayesian lr (bbb) and sbert gp. 5 overall, the incorrect predictions (> 1 from the true score) by bbb have a much smaller variance compared to those predicted by gp. for correct predictions (≤ 1 of the true score), bbb has a higher variance than for incorrect predictions, which is counter-intuitive. though the std for sbert gp regression on correct predictions is much larger than bbb, it's slightly less than that for incorrect ones. this fits the expectation that when a model is good at uncertainty prediction, the model should be more confident for correct predictions than incorrect ones. examples where both models are correct and incorrect are presented in the near-zero variance of bbb (0.005 on average) results in infinite nlpd because of the element (t i -m i ) 2 v i in the nlpd formula. larger shp of gp tends to produce smaller nlpd in spite of being under-confident on correct cases-the variance of 1.57 is much larger than the true gap of 0.01. so nlpd is not a perfect metric, favouring under-confident models. we therefore suggest a metric priority order of r, cal, nlpd and shp. impact of sentence embedding: the quality of sentence embeddings is critical for uncertainty training, affecting not only the correlation, but also the uncertainty metrics. instead of sbert, we also experimented with simcse, the current state-of-the-art sentence encoder @xcite . we train three pipeline models with sts-b training data based on sup-simcse-bert-baseuncased, using the same settings as the first row of high-disagreement label detection: a natural question to ask in the instance selection is what types of instances are selected and discarded, and how this correlates with the underlying label uncertainty in the data. when models are well-calibrated, the predicted variance will reflect the true label uncertainty, both aleatoric and epistemic. as such, if we select instances with smaller variance, we are effectively filtering out instances with higher inherent label uncertainty, as should be reflected in the labels assigned by independent annotators. we verify this hypothesis below. we apply the model fine-tuned on sts-b over biosses and ebmsass (1000 instances each), for which five raw annotations for each instance can be accessed to approximate an empirical label distribution. kl-divergence (kl) is used to measure the distance between the predicted and empirical probability. in do large-variance instances selected by strategies in section 4.2 overlap with high-disagreement instances? without a ground truth of highdisagreement annotations, they are identified by two steps iteratively: (1) select labels whose std is greater than α, beginning from 0.3; and (2) manually check whether for all selected instances, at least two out of the five annotations differ from the others @xmath3.0; if not α+=0.1, otherwise end. this results in 137 and 31 label disagreements when α = 0.5 and 0.4, for ebmsass and biosses, respectively. using bert lr mc-dropout, a learned threshold of τ = 0.162 results in @xmath4.48, @xmath5.28 at high-disagreement label detection on ebmsass. for biosses, τ = 0.1 leads to @xmath6.37, @xmath7.48. under convlr mc, ebmsass has @xmath8.46, @xmath9.31 as τ = 0.124; biosses: τ = 0.157 with @xmath10.45, @xmath11.48. as such, high-disagreement labels can be detected by the large-variance criterion, obtaining @xmath12.44, @xmath13.39 on average. this is not good as a binary classifier, since regarding all instances as the majority-class ''clean'' performs better. but in our context, it is effective as a data augmentation strategy-selecting clean examples from an out-of-domain corpus. detecting noisy labels is not just a binary classification task requiring high accuracy, but critical to recognize and filter noisy instances from a whole training corpus, even at the cost of removing clean labels. ## conclusion we comprehensively investigated a range of uncertainty estimation methods over different regression tasks, using pre-trained language models. bayesian linear regression and sparse gaussian process regression based on fixed features obtain lower calibration error and nlpd compared with fine-tuning large-capacity deep networks end-to-end, but are inferior in terms of correlation. when embeddings are sufficiently expressive, they are comparable in performance to deep uncertainty models. to reduce uncertainty resulting from noisy labels and limited labeled data in specific domains, we proposed a simple instance selection method based on uncertainty model predictive confidence. this approach demonstrated consistent performance improvements on three regression tasks in both self-training and active-learning settings, underscoring its effectiveness and generalizability.