As Little as Possible, as Much as Necessary: Detecting Over- and Undertranslations with Contrastive Conditioning (2022)

Abstract: Omission and addition of content is a typical issue in neural machine translation. We propose a method for detecting such phenomena with off-the-shelf translation models. Using contrastive conditioning, we compare the likelihood of a full sequence under a translation model to the likelihood of its parts, given the corresponding source or target sequence. This allows to pinpoint superfluous words in the translation and untranslated words in the source even in the absence of a reference translation. The accuracy of our method is comparable to a supervised method that requires a custom quality estimation model.

## introduction neural machine translation (nmt) is susceptible to coverage errors such as the addition of superfluous target words or the omission of important source content. previous approaches to detecting such errors make use of reference translations @xcite or employ a separate quality estimation (qe) model trained on synthetic data for a language pair @xcite . in this paper, we propose a reference-free algorithm based on hypothetical reasoning. our premise is that a translation has optimal coverage if it uses as little information as possible and as much information as necessary to convey the source sequence. therefore, an addition error means that the source would be better conveyed by a translation containing less information. conversely, an omission error means that the translation would be more adequate for a less informative source sequence. adapting our contrastive conditioning approach @xcite , we use probability scores of nmt models to approximate this concept of coverage. we create parse trees for both the source sequence and the translation, and treat their constituents as units of information. omission errors are detected by systematically deleting constituents from the source and by estimating the probability of the translation conditioned on such a partial source sequence. if the probability score is higher than when the translation is conditioned on the full source, the deleted constituent might have no counterpart in the translation (figure 1 ). we apply the same principle to the detection of addition errors by swapping the source and the target sequence. when comparing the detected errors to human annotations of coverage errors on the segment level @xcite , our approach surpasses a supervised qe baseline that was trained on a large number of synthetic coverage errors. human raters find that word-level precision is higher for omissions than additions, with 39% of predicted error spans being precise for english-german translations, and 20% for chinese-english. false positive predictions can occur especially in cases where the translation has different syntax than the source. we believe our algorithm could be a useful aid whenever humans remain in the loop, for example in a post-editing workflow. we release the code and data to reproduce our findings, including a large-scale dataset of synthetic coverage errors in english-german and chinese-english machine translations. foot_0 2 related work coverage errors in nmt addition and omission of target words have been observed by human evaluation studies in various languages, with omission as the more frequent error type @xcite . they are included as typical translation issues in the multidimensional quality metrics (mqm) framework @xcite . addition is defined as an accuracy issue where the target text includes text not present in the source, and omission is defined as an accuracy issue where content is missing from the translation but is present in the source. 2 freitag et al. (2021) used mqm to manually re-annotate english-german and chinese-english machine translations submitted to the wmt 2020 news translation task @xcite . their findings confirm that state-of-the-art nmt systems still erroneously add and omit target words, and that omission occurs more often than addition. similar patterns can be found in english-french machine translations that have been annotated with fine-grained mqm labels for the document-level qe shared task @xcite @xcite . detecting and reducing coverage errors while reference-based approaches include measuring the n-gram overlap to the reference @xcite and analyzing word alignment to the source @xcite , this work focuses on the reference-free detection of coverage errors. previous work has employed custom qe models trained on labeled parallel data. for example, @xcite insert synthetic hallucinations and train a transformer to predict the inserted spans. similarly, @xcite train a qe model on synthetically noisy translations. in this paper, we propose a method that is based on off-the-shelf nmt models only. other related work has focused on improving coverage during decoding or training, for example via attention @xcite @xcite . more recently, @xcite found that contrastive fine-tuning on references with synthetic omissions reduces coverage errors produced by an nmt system. ## approach contrastive conditioning properties of a translation can be inferred by estimating its probability conditioned on contrastive source sequences @xcite . for example, if a certain translation is more probable under an nmt model when conditioned on a counterfactual source sequence, the translation might be inadequate. application to omission errors figure 1 illustrates how contrastive conditioning can be directly applied to the detection of omission errors. we construct partial source sequences by systematically deleting constituents from the source. if the probability score of the translation (average token logprobability) is higher when conditioned on such a partial source, the deleted constituent is taken to be missing from the translation. to compute the probability score for a translation y given a source sequence x, we sum up the log-probabilities for every target token and normalize the sum by the number of target tokens: application to addition errors we apply the same method to addition detection, but swap the source and target languages. namely, we use an nmt model for the reverse translation direction, and we score the source sequence conditioned on the full translation and a set of partial translations. potential error spans in its most basic form, our algorithm does not require any linguistic resources apart from tokenization. for a source sentence of n tokens one could create n partial source sequences with the ith token deleted. however, such an approach would rely on a radical assumption of compositionality, treating all tokens as independent constituents. we thus propose to extract potential error spans from parse trees, specifically from dependency trees predicted by universal dependency parsers @xcite , which are widely available. this allows (a) to skip function words and (b) to include a reasonable number of multiword spans in the set of potential error spans. formally, we consider word spans that satisfy the following conditions: ## experimental setup in this section we describe the data and tools that we use to implement and evaluate our approach. scoring model we use mbart50 @xcite , which is a sequence-to-sequence transformer pre-trained on monolingual corpora in many languages using the bart objective @xcite that was fine-tuned on english-centric multilingual mt in 50 languages. sequence-level probability scores are computed by averaging the log-probabilities of all target tokens. we use the one-to-many mbart50 model if english is the source language, and the many-to-one model if english is the target language. error spans we use stanza @xcite for dependency parsing, a neural pipeline for various languages trained on data from universal dependencies (de @xcite . we make use of universal part-of-speech tags (upos) to define ditioned on the source. however, the scores might be confounded by a lack of fluency in the partial translations. the full translation contains an addition error with regard to the partial source, and the partial translation contains an omission error with regard to the original source sequence. ## limitations and future work we hope that the automatic detection of coverage errors could be an aid to translators and posteditors, given that manually detecting such errors is tedious. our results on omissions are encouraging, and user studies are recommended in order to validate the usefulness of the predictions to practitioners. further work needs to be done to improve the detection of additions, of which the real-world data contain few examples. higher accuracy would be necessary for word-level qe to be helpful @xcite , and so with regard to detecting addition errors, the practical utility of both the baseline and of our approach remains limited. inference time should also be discussed. in appendix c we perform a comparison, finding that on a long sentence pair contrastive conditioning can take up to ten times longer than a forward pass of the baseline. however, this is still a fraction of the time needed for generating a translation in the first place. in addition, restricting the potential error spans that are considered could further improve efficiency. ## conclusion we have proposed a reference-free method to automatically detect coverage errors in translations. derived from contrastive conditioning, our method relies on hypothetical reasoning over the likelihood of partial sequences. since any off-the-shelf nmt model can be used to estimate conditional likelihood, no access to the original translation system or to a quality estimation model is needed. evaluation on real machine translations shows that our approach outperforms a supervised baseline in the detection of omissions. future work could address the low precision on addition errors, which are relatively rare in the datasets we used for evaluation.