On the Inference Calibration of Neural Machine Translation (2020)

Abstract: Confidence calibration, which aims to make model predictions equal to the true correctness measures, is important for neural machine translation (NMT) because it is able to offer useful indicators of translation errors in the generated output. While prior studies have shown that NMT models trained with label smoothing are well-calibrated on the ground-truth training data, we find that miscalibration still remains a severe challenge for NMT during inference due to the discrepancy between training and inference. By carefully designing experiments on three language pairs, our work provides in-depth analyses of the correlation between calibration and translation performance as well as linguistic properties of miscalibration and reports a number of interesting findings that might help humans better analyze, understand and improve NMT models. Based on these observations, we further propose a new graduated label smoothing method that can improve both inference calibration and translation performance.

## introduction calibration requires that the probability a model assigns to a prediction (i.e., confidence) equals to the correctness measure of the prediction (i.e., accuracy). calibrated models are important in userfacing applications such as natural language processing @xcite and speech recognition @xcite , in which one needs to assess the confidence of a prediction. for example, in computer-assisted translation, a calibrated machine translation model is able to tell a user when the model's predictions are likely to be incorrect, which is helpful for the user to correct errors. ## related work calibration on classification calibration on classification tasks has been studied for a long history in the statistics literature, including platt scaling @xcite , isotonic regression @xcite and many other methods for non-binary classification @xcite @xcite . for modern deep neural networks, @xcite demonstrated bush held a talk with sharon in israel . bush attended a public talk with sharon . that recent advances in training and model architecture have strong effects on the calibration. szegedy et al. (2016) propose the label smoothing technique which can effectively reduce the calibration error. ding et al. (2019) extend label smoothing to adaptive label regularization. calibration on structured prediction different from classification tasks, most natural language processing (nlp) tasks deal with complex structures @xcite . @xcite verified the finding of @xcite in nlp tasks on loglinear structured models. for nmt, some works directed their attention to the uncertainty in prediction @xcite , @xcite studied the calibration of several nmt models and found that the end of a sentence is severely miscalibrated. @xcite investigated the effect of label smoothing, finding that nmt models are well-calibrated in training. different from previous works, we are interested in the calibration of nmt models in inference, given that the training and inference are discrepant for standard nmt models @xcite . ## linguistic properties of miscalibration in this section, we investigate the linguistic properties of miscalibrated tokens in nmt outputs. we explore the following five types of properties: frequency, position, fertility, syntactic roles, and word granularity. frequency is generally related to miscalibration; position, fertility, and word granularity are three factors associated with structured prediction; syntactic roles or linguistic roles may vary across language pairs. the results in this section are reported on the held-out set by the retrained model. relative change we use the relative change of the proportion of a certain category of tokens to quantify to what extent they suffer from the under/over-estimation. for instance, in the zh-en task, high-frequency tokens account for 87.6% on the whole held-out set, and among over-estimated tokens, high-frequency tokens account for 77.3%, thus for over-estimation the relative change of highfrequency tokens is (77.3-87.6)/87.6=-11.76% in zh-en. accordingly, the value of the red rectangle of zh-en is -11.76% in figure 4a . positive relative change denotes that a certain type of linguistic property accounts more in miscalibrated predictions than in all the predictions, suggesting this type of linguistic property suffers from the miscalibration problem. similarly, negative relative change suggests that a certainty type of linguistic property is less likely to be impaired by the miscalibration problem. ## conclusion although nmt models are well-calibrated in training, we observe that they still suffer from miscalibration during inference because of the discrepancy between training and inference. through a series of in-depth analyses, we report several interesting findings which may help to analyze, understand and improve nmt models. we revisit recent advances and find that label smoothing and dropout play key roles in calibrating modern nmt models. we further propose graduated label smoothing that can reduce the inference calibration error effectively. finally, we find that increasing model size can negatively affect the calibration of nmt models and this can be alleviated by only enlarging the encoder. as well-calibrated confidence estimation is more likely to establish trustworthiness with users, we plan to apply our work to interactive machine translation scenarios in the future.