Learning Confidence for Transformer-based Neural Machine Translation (2022)

Abstract: Confidence estimation aims to quantify the confidence of the model prediction, providing an expectation of success. A well-calibrated confidence estimate enables accurate failure prediction and proper risk measurement when given noisy samples and out-of-distribution data in real-world settings. However, this task remains a severe challenge for neural machine translation (NMT), where probabilities from softmax distribution fail to describe when the model is probably mistaken. To address this problem, we propose an unsupervised confidence estimate learning jointly with the training of the NMT model. We explain confidence as how many hints the NMT model needs to make a correct prediction, and more hints indicate low confidence. Specifically, the NMT model is given the option to ask for hints to improve translation accuracy at the cost of some slight penalty. Then, we approximate their level of confidence by counting the number of hints the model uses. We demonstrate that our learned confidence estimate achieves high accuracy on extensive sentence/word-level quality estimation tasks. Analytical results verify that our confidence estimate can correctly assess underlying risk in two real-world scenarios: (1) discovering noisy samples and (2) detecting out-of-domain data. We further propose a novel confidence-based instance-specific label smoothing approach based on our learned confidence estimate, which outperforms standard label smoothing.

## introduction confidence estimation has become increasingly critical with the widespread deployment of deep neural networks in practice @xcite . it aims to measure the model's confidence in the prediction, showing when it probably fails. a calibrated confidence estimate can accurately identify failure, further measuring the potential risk induced by noisy samples and out-of-distribution data prevalent in real scenarios @xcite . unfortunately, neural machine translation (nmt) is reported to yield poor-calibrated confidence estimate @xcite , which is common in the application of modern neural networks @xcite . it implies that the probability a model assigns to a prediction is not reflective of its correctness. even worse, the model often fails silently by providing high-probability predictions while being woefully mistaken @xcite . we take figure 1 as an example. the mistranslations are produced with high probabilities (dark green blocks in the dashed box), making it problematic to assess the quality based on prediction probability when having no access to references. the confidence estimation on classification tasks is well-studied in the literature @xcite . yet, researches on structured generation tasks like nmt is scarce. existing researches only study the phenomenon that the generated probability in nmt cannot reflect the accuracy @xcite , while little is known about how to establish a well-calibrated confidence estimate to describe the predictive un-certainty of the nmt model accurately. to deal with this issue, we aim to learn the confidence estimate jointly with the training process in an unsupervised manner. inspired by ask for hints (devries and taylor, 2018), we explain confidence as how many hints the nmt model needs to make a correct prediction. specifically, we design a scenario where ground truth is available for the nmt model as hints to deal with tricky translations. but each hint is given at the price of some penalty. under this setting, the nmt model is encouraged to translate independently in most cases to avoid penalties but ask for hints to ensure a loss reduction when uncertain about the decision. more hints mean low confidence and vice versa. in practice, we design a confidence network, taking multi-layer hidden states of the decoder as inputs to predict the confidence estimate. based on this, we further propose a novel confidence-based label smoothing approach, in which the translation more challenging to predict has more smoothing to its labels. recall the example in figure 1 . the first phrase "a figure who loves to play" is incorrect, resulting in a low confidence level under our estimation. we notice that the nmt model is also uncertain about the second expression "a national class actor", which is semantically related but has inaccurate wording. the translation accuracy largely agrees with our learned confidence rather than model probabilities. we verify our confidence estimate as a wellcalibrated metric on extensive sentence/word-level quality estimation tasks, which is proven to be more representative in predicting translation accuracy than existing unsupervised metrics @xcite . further analyses confirm that our confidence estimate can precisely detect potential risk caused by the distributional shift in two real-world settings: separating noisy samples and identifying out-of-domain data. the model needs more hints to predict fake or tricky translations in these cases, thus assigning them low confidence. additionally, experimental results show the superiority of our confidence-based label smoothing over the standard label smoothing technique on different-scale translation tasks (wmt14 en⇒de, nist zh⇒en, wmt16 ro⇒en, and iwslt14 de⇒en). the contributions of this paper are three-fold: ## background in this section, we first briefly introduce a mainstream nmt framework, transformer @xcite , with a focus on how to generate prediction probabilities. then we present an analysis of the confidence miscalibration observed in nmt, which motivates our ideas discussed afterward. ## learning to estimate confidence a well-calibrated confidence estimate should be able to tell when the nmt model probably fails. ideally, we would like to learn a measure of confidence for each target-side translation, but this remains a thorny problem in the absence of ground truth for confidence estimate. inspired by ask for hints (devries and taylor, 2018) on the image classification task, we define confidence as how many hints the nmt model needs to produce the correct translation. more hints mean low confidence, and that is a high possibility of failure. motivation. we assume that the nmt model can ask for hints (look at ground-truth labels) during training, but each clue comes at the cost of a slight penalty. intuitively, a good strategy is to indepen-dently make the predictions that the model is confident about and then ask for clues when the model is uncertain about the decision. under this assumption, we approximate the confidence level of each translation by counting the number of hints used. to enable the nmt model to ask for hints, we add a confidence estimation network (connet) in parallel with the original prediction branch, as shown in figure 3 . the connet takes hidden states of the decoder at t-th step (h t ) as inputs and predicts a single scalar between 0 and 1. to offer the model "hints" during training, we adjust softmax prediction probabilities by interpolating the ground truth probability distribution y t (denoted as a one-hot vector) into the original prediction. the degree of interpolation is decided by the generated confidence c t : the translation loss is calculated using modified prediction probabilities. to prevent the model from minimizing the loss by always setting c @xmath0, we add a log penalty to the loss function. the final loss is the sum of the translation loss and the confidence loss, which is weighted by the hyper-parameter λ: under this setting, when @xmath1, we can see that p ′ → p and l @xmath2, which is equal to a standard training procedure. in the case where @xmath3, we see that p ′ → y (the model obtains correct labels). in this scenario, l nmt would approach 0, but l conf becomes very large. thus, the model can reduce the overall loss only when it successfully predicts which outputs are likely to be correct. ## experiments this section first exhibits empirical studies on the quality estimation (qe) task, a primary application of confidence estimation. then, we present experimental results of our confidence-based label smoothing, an extension of our confidence estimate to better smoothing in nmt. ## analysis confidence estimation is particularly critical in realworld deployment, where noisy samples and out-ofdistribution data are prevalent @xcite . given those abnormal inputs, neural network models are prone to be highly confident in misclassification @xcite . thus, we need an accurate confidence estimate to detect potential failures caused by odd inputs by assigning them low confidence. this section explores whether our confidence estimate can accurately measure risk under those two conditions. ## related work the task of confidence estimation is crucial in realworld conditions, which helps failure prediction @xcite and out-of-distribution detection @xcite @xcite . this section reviews recent researches on confidence estimation and related applications on quality estimation for nmt. ## conclusion in this paper, we propose to learn confidence estimates for nmt jointly with the training process. we demonstrate that learned confidence can better indicate translation accuracy on extensive sentence/word-level qe tasks and precisely measures potential risk induced by noisy samples or out-of-domain data. we further extend the learned confidence estimate to improve smoothing, outper-forming the standard label smoothing technique. as our confidence estimate outlines how much the model knows, we plan to apply our work to design a more suitable curriculum during training and post-edit low-confidence translations in the future.