Measuring Uncertainty in Neural Machine Translation with Similarity-Sensitive Entropy (2024)

Abstract: Uncertainty estimation is an important diagnostic tool for statistical models, and is often used to assess the confidence of model predictions. Previous work shows that neural machine translation (NMT) is an intrinsically uncertain task where there are often multiple correct and semantically equivalent translations, and that well-trained NMT models produce good translations despite spreading probability mass among many semantically similar translations. These findings suggest that popular measures of uncertainty based on token- and sequence-level entropies which measure surface form diversity may not be good proxies of the more useful quantity of interest, semantic diversity. We propose to adapt similarity-sensitive Shannon entropy (S3E), a concept borrowed from theoretical ecology, for NMT. By demonstrating significantly improved correlation between S3E and task performance on quality estimation and named entity recall, we show that S3E is a useful framework for measuring uncertainty in NMT.

## introduction uncertainty estimation has a wide range of applications in neural machine translation (nmt), including unsupervised quality estimation @xcite , semi-supervised learning @xcite , curriculum learning @xcite , active learning @xcite , interactive translation @xcite , and more. many different measures exist for capturing uncertainty, each developed for the application at hand. nmt is an intrinsically uncertain task, where a source sentence can have multiple correct translations which are equivalent in meaning @xcite . even large nmt models are known in practice to spread probability mass across a large number of translations. but the diffusive quality of the nmt distribution is not necessarily a problem in theory or in practice; in theory, the true data distribution may be diffuse, hence a perfect model will also be diffuse. in practice, high-probability translations are highly semantically similar to each other, and model probability correlates reasonably well with actual quality @xcite . nmt models are generally evaluated on their ability to generate the desired semantics irrespective of lexical form @xcite , hence the uncertainty measure used to assess the confidence of a model prediction should also measure semantic diversity rather than lexical diversity. nmt distributions are known to be highly diverse over surface forms, i.e. token sequences, but diversity over token sequences does not necessarily reflect semantic diversity. we therefore suspect that surface form uncertainty measures over the model distribution such as token-or sequence-level entropy would not be good measurements of model confidence compared to ones that accounts for semantic similarity across sequences. with this motivation, we propose to adapt similarity-sensitive shannon entropy (s3e) @xcite to measure semantic uncertainty in nmt. s3e was originally proposed in theoretical ecology to quantify biodiversity while accounting for species similarity, but it is a general framework that permits flexibility in defining the similarity function, and thus has broad applicability beyond ecology. we adapt s3e to nmt tasks by specifying appropriate similarity functions. we also show how the s3e framework relates to and generalizes previous work on uncertainty estimation, and present practical methods for estimating s3e efficiently and accurately for nmt. in quality estimation (qe) experiments, we estimate s3e using embeddings from models pretrained on large amounts of data, and show that this has higher correlation with translation quality than previously used similarityinsensitive uncertainty measures. further, to illus-trate the flexibility of s3e and the importance of matching the similarity function to the task, we perform a named entity recall task, where we find that the best correlation with task performance is achieved when specifying a similarity function focusing exclusively on named entities. ## similarity-sensitive entropy in theoretical ecology, similarity-sensitive measures of biodiversity which allow for flexible specifications of similarity have been studied extensively @xcite . the similarity-sensitive shannon entropy (s3e), originally proposed by @xcite and for which @xcite provides a comprehensive treatment, is defined as: the difference between s3e and shannon entropy (se) is that the negative surprisal of an event is not log probability but log of the expected similarity between the outcome and all other outcomes. we call this the similarity-sensitive surprisal (sss). intuitively, outcomes are less surprising or informative if they are similar to other outcomes. we note a few desirable properties of s3e: in other words, there is no uncertainty if all outcomes are the same. proofs for these properties and of all theoretical details in this work are in the appendix. the close relation between se and s3e means that by comparing the two empirically, we study the impact of the choice of s on the usefulness of the uncertainty measure. ## experiments our experiments are conducted on english-german (en-de), estonian-english (et-en), and nepali-english (ne-en), representing high, medium, and low resource language pairs respectively. we use pre-trained translation models for all experiments. for en-de, we use the ensemble translation model from @xcite foot_1 . for et-en and ne-en, we use the many-to-one multilingual model from @xcite foot_2 . in sections 4.1.1 and 4.4, our s3e estimation procedure requires o(n 2 ) calls to the semantic similarity function. in order to keep a reasonable runtime, we use sentence embedding models where similarity is computed with cosine distance. this way, the expensive embedding step is linear time, while only the faster cosine similarity computation is quadratic time. for german sentence embedding, we use a multilingual sbert model foot_3 . for english, we use supervised simcse 5 . cosine similarity ranges from -1 to 1. to create a valid s3e similarity function s, we replace negative values with 0 following (vor der brück and pouly, 2019). negative cosine similarity is rare in practice and may be caused by antonymy, which we ignore. let f be a sequence embedding model. then: as a baseline similarity function, we use the sacre-bleu @xcite implementation of chrf++ @xcite with default settings and normalize the range to [0, 1]. to obtain model predictions, we use beam search with beam size 5 for all language pairs. whenever random samples are employed, we obtain 128 samples for each instance with ϵ-sampling @xcite with ϵ = 0.02, which was shown by @xcite to perform well in mbr decoding. in all experiments, we first tune the s3e similarity scaling parameter α for best performance on a validation set and only report results on the test set. we search for the optimal α over all integers in [-1, 10]. in tables 1 , 2 , and 3, the optimal α found in validation and used in test is displayed in parentheses beside relevant results. our code is publicly available foot_5 . ## conclusion we believe that s3e is a useful framework for understanding, comparing, and developing measures of uncertainty for tasks in nlp and beyond. important steps forward for s3e are: 1) the development of faster and/or more accurate similarity functions, 2) the application of s3e to parts of the nmt training pipeline, such as semi-supervised learning and active learning, 3) the application of s3e to other conditional language generation tasks, and 4) extensions to theory which explicitly model other sources of uncertainty, such as epistemic uncertainty.