Contrastive Conditioning for Assessing Disambiguation in MT : A Case Study of Distilled Bias (2021)

Abstract: Lexical disambiguation is a major challenge for machine translation systems, especially if some senses of a word are trained less often than others. Identifying patterns of overgeneralization requires evaluation methods that are both reliable and scalable. We propose contrastive conditioning as a reference-free black-box method for detecting disambiguation errors. Specifically, we score the quality of a translation by conditioning on variants of the source that provide contrastive disambiguation cues. After validating our method, we apply it in a case study to perform a targeted evaluation of sequence-level knowledge distillation. By probing word sense disambiguation and translation of gendered occupation names, we show that distillation-trained models tend to overgeneralize more than other models with a comparable BLEU score. Contrastive conditioning thus highlights a side effect of distillation that is not fully captured by standard evaluation metrics. Code and data to reproduce our findings are publicly available.

## introduction erroneous disambiguation of words makes translations inadequate and can even constitute a form of bias when it occurs systematically. however, detecting disambiguation errors in machine translation (mt) is a non-trivial task. previous work has focused on automatic post-hoc analysis of translations @xcite , but rules of what makes a disambiguation correct or incorrect tend to be imprecise. while contrastive evaluation @xcite eliminates the need for post-hoc analysis by scoring pre-defined pairs of hypotheses, such probability estimates cannot be obtained from black-box systems, e.g., from commercial apis that only return a 1-best translation to the user. in addition, contrastive hypotheses need to be carefully crafted for every target language of interest. we propose contrastive conditioning as a scalable black-box alternative for evaluating disambiguation in machine translation. the evaluated translations are paired with contrastive source sequences and are then scored by a white-box translation model. the contrastive sources are variants of the original source, slightly modified to provide a stronger disambiguation cue. for example, consider a model that translates the english source 'doctor' into german as 'ärztin' (female doctor). this translation will receive a better score when conditioned on the source 'female doctor' than on 'male doctor', indicating that it is a feminine form. given sufficient disambiguation cues, the whitebox translation model can thus serve as an evaluator for the original translation. since the contrastive sources are written in the source language, contrastive conditioning does not rely on references in the target language. this makes it easier to scale the evaluation across multiple target languages. in addition, the method is reliable compared to alternative evaluation methods, according to our human validation. the assistant asked the doctor if she needs any help. der assistent fragte die ärztin, ob sie hilfe brauche. die assistentin fragte den arzt, ob sie hilfe brauche. we ought to have avocados as a starter. wir müssen avocados als vorspeise haben. wir müssen avocados als anlasser haben. ## discussion contrastive conditioning is a new protocol for evaluating and comparing mt systems with regard to word sense disambiguation. in our analysis we build on established test sets, but have replaced the standard pattern-matching analysis with contrastive conditioning. a human validation study, with english as a source language, showed that the approach is reliable, especially if the samples are weighted according to evaluator confidence. an advantage of our approach over patternmatching is that it can process any potential translation, not just translations containing pre-defined lemmas or translations that have certain morphosyntactic properties. furthermore, an advantage over conventional contrastive evaluation methods is that the decoding mode of the evaluated model does not need to be constrained. thus, black-box systems, for example apis of commercial systems, can be evaluated too. finally, test sets can remain reference-free, and we even believe that neither strong assumptions nor deeper expertise of a target language are strictly required to perform contrastive conditioning (even though in this paper, we put forward some linguistically informed arguments to motivate our approach). a limitation of contrastive conditioning is that a disambiguation cue needs to be provided in the source language. for error types other than disambiguation, such a cue might be difficult to create. in this paper, we have built on purely textual cues, which enabled us to use an off-the-shelf translation model for scoring. linguistic input features @xcite could provide an alternative disambiguation cue in future work. our case study of knowledge distillation, which is based on contrastive conditioning, shows that seqkd can lead to lexical overgeneralization, and to a loss of adequacy in disambiguation that is generally not captured by bleu. ## conclusion in order to evaluate mt models on disambiguation, we have devised a novel evaluation method, contrastive conditioning. it allows for a reference-free, black-box evaluation of mt models with respect to disambiguation, requiring only that a strong disambiguation cue can be provided in contrastive sources. based on this evaluation method, we have presented a case study of translation models trained with sequence-level knowledge distillation. focusing on the issue of lexical overgeneralization in word sense disambiguation, we have tested the models on word sense disambiguation and the translation of gendered occupations. our results indicate that sequence-level knowledge distillation can amplify existing imbalances in the training data, and typically leads to an increased overgeneralization bias. we encourage future work to develop methods that reap the benefits of knowledge distillation with minimal increases in bias.