Reducing Disambiguation Biases in NMT by Leveraging Explicit Word Sense Information (2022)

Abstract: Recent studies have shed some light on a common pitfall of Neural Machine Translation (NMT) models, stemming from their struggle to disambiguate polysemous words without lapsing into their most frequently occurring senses in the training corpus. In this paper, we first provide a novel approach for automatically creating high-precision sense-annotated parallel corpora, and then put forward a specifically tailored fine-tuning strategy for exploiting these sense annotations during training without introducing any additional requirement at inference time. The use of explicit senses proved to be beneficial to reduce the disambiguation bias of a baseline NMT model, while, at the same time, leading our system to attain higher BLEU scores than its vanilla counterpart in 3 language pairs.

## introduction translating a sentence requires the underlying meaning to be captured and then expressed in the target language. nonetheless, only little attention has been devoted to studying the actual capabilities of neural machine translation (nmt) approaches of modeling different senses of ambiguous words, with recent work showing that systems tend to be biased towards the most frequent meanings found within the training corpus @xcite . this phenomenon is hard to measure through classical evaluation metrics, such as the bleu score @xcite , as they often rely on word-matching heuristics that fail to capture the disambiguation capabilities of the evaluated systems. therefore, several efforts have been recently devoted to shed some light and create test beds @xcite @xcite to challenge nmt models. results show that these models still struggle to deal with highly polysemous words, especially when used to express least frequent senses. for example, given the sentence "the energy comes from a distant plant.", both google translate and deepl disambiguate foot_0 plant to its sense of organism when translating into italian, and produce the following incorrect sentence "l'energia proviene da una pianta lontana.", rather than "l'energia proviene da un impianto lontano.", where impianto is the translation for the factory meaning of plant. this suggests that, even when adequate context is provided (energy should be enough to correctly infer the right sense of plant), state-of-the-art models might still be biased towards the most frequent meanings found within training data. some recent studies have explored how to leverage explicit sense information within nmt models @xcite @xcite . nevertheless, including such information is not trivial for three main reasons: i) sense-tagged parallel data is scarce; ii) word sense disambiguation (wsd) systems have not been accurate enough until very recently @xcite ; and iii) how explicit senses should be incorporated within neural models is not straightforward. in this paper, we first introduce a novel approach to make up for the paucity of sense annotations in parallel corpora, leveraging a multilingual wsd system to tag parallel sentences and refine its predictions by means of cross-lingual word alignments and information from a multilingual knowledge base. then, we fine-tune our baseline models on our sense-tagged corpora via a specifically designed loss function, allowing the injection of wordlevel semantics into the architecture. we evaluate our approach on standard and challenge test sets, showing that it does indeed improve translation accuracy and mitigates the most frequent sense bias. to summarize, our contributions are manifold: we make all the generated datasets, the code of the model and for the experiments available at https://github.com/sapienzanlp/ reducing-wsd-bias-in-nmt . ## related work word sense disambiguation was first formulated as a computational task by @xcite in the context of machine translation. the two fields then followed parallel paths, with more or less successful attempts over the years to join them back together @xcite @xcite . indeed, while @xcite reported negative results when trying to integrate the prediction of a supervised wsd approach into a statistical machine translation (smt) model, the same authors, two years later, successfully improved the performance of a phrase-based smt approach by leveraging a new phrase-based wsd model @xcite . more recently, @xcite and @xcite proposed systems that successfully leverage sense information in nmt models, although they introduced a heavy requirement, i.e., that of disambiguating the ambiguous words in the sentence prior to generating a translation, which makes them unfeasible in many real-world settings. lately, contextualized word embeddings have been employed to produce additional back-translated parallel training data via mining sense-specific target sentences, in order to improve handling of infrequent senses @xcite . nevertheless, the proper treatment of lexical ambiguity is still an open problem, with neural models struggling to translate least frequent senses and often relying on spurious correlations among words @xcite @xcite . thus, the disambiguation bias topic has received renewed interest, and several benchmarks have been introduced in the most recent years with the goal of directly measuring the extent to which neural architectures are able to capture word semantics. one of the first of this kind was contrawsd (rios @xcite . in this first attempt to evaluate wsd capabilities of nmt models, the authors built an adversarial test set where source sentences containing an ambiguous word were associated with a correct translation and several incorrect alternatives. these latter were built by replacing the reference translation for the ambiguous word with the translation of one of its other possible meanings. the task measured whether a model ranked the correct translation higher, i.e., it assigned it a higher probability than the adversarial ones. this study provided evaluation data for two language pairs only, i.e., @xmath0, and within a few years it became outdated as modern nmt models could easily attain high performances @xcite . thus, mucow @xcite took things a step further and leveraged babelnet @xcite -a large multilingual knowledge base -and sense embeddings @xcite in order to automatically create adversarial translations for five language pairs while also increasing the difficulty of the task itself; however, the fully automatic nature of these challenge sets made them noisy and prone to containing irrelevant challenge samples. more recently, @xcite proposed two challenge sets for the @xmath1, one measuring the model sensitivity to most frequent senses and the other estimating, through adversarial injections, its susceptibility to changing a correct sense to a wrong one. in contrast to previous studies, these challenge sets were based on correlations among words in the training set and relied on manually-refined sense clusters, providing an excellent test bed for measuring semantic bias. finally, @xcite proposed dibimt, the first fully manually annotated test set for measuring the disambiguation bias of neural machine translation models, covering five language combinations, namely, from english to german, spanish, italian, russian and chinese. in their work, the authors showed that open neural models still exhibit strong semantic biases towards frequent senses, confirming once again the suspicions about this under-explored issue. despite all the effort made in putting forward challenging sets of data to test wsd capabilities of nmt models, to the best of our knowledge, only a few approaches @xcite have been proposed to mitigate this issue, and none of these is effective with modern transformer-based architectures. furthermore, while parallel corpora have been exploited to produce sense annotations in the past @xcite , they were built by utilizing outdated disambiguation approaches that have recently been surpassed by more advanced neural architectures. indeed, the word sense disambiguation field has received much attention in the last few years, with several supervised approaches @xcite @xcite and sense embedding models @xcite @xcite performing close to the upper bound limit of the inter-annotator agreement, which finally makes them feasible for inclusion in other downstream tasks, e.g., machine translation. thus, differently from previous studies in the literature, we focus on closing the gap between these two fields, i.e., neural machine translation and word sense disambiguation, by putting the recent advances in wsd at the service of nmt models. we propose a novel approach, similar to that introduced in @xcite , for creating high-quality sense-annotated parallel corpora, and we use this semantic information to regularize an nmt model, making it less biased and capable of producing higher-quality translations. ## reducing the disambiguation bias in nmt neural machine translation models are typically trained end-to-end to produce a target translation given a source sentence and, thus, they can only rely on the input context to resolve the ambiguity of polysemous words therein. being pattern recognition algorithms at heart, these models fall prey to the inherent bias carried by the frequency of co-occurrence of words within parallel sentences, and thus tend to disambiguate words to the sense they most frequently encountered during training, even when the sentence does provide enough context to identify the correct sense. at the same time, word sense disambiguation models, i.e., models specialized in associating a word in context with one of the meanings within a given sense inventory, have recently displayed remarkable results across different benchmarks and languages @xcite . the time may now therefore be ripe for them to be successfully included into downstream applications such as neural machine translation. however, data that would allow these two worlds to be brought together, i.e., parallel corpora where words are associated with semantic labels, are currently still produced automatically by leveraging outdated approaches to wsd @xcite . in what follows, we first provide some preliminary information about resources and tools that we employ in our method ( § 3.1); then, we introduce a new approach for automatically annotating tokens within parallel sentences with sense annotations, i.e., labels explicitly defining their meanings ( § 3.2); finally, we propose a fine-tuning objective for leveraging such annotations in order to mitigate the sense bias while also improving the translation quality overall ( § 3.3). the intuition behind our work is that fixed sense labels describing word senses would help nmt models better encode the underlying meaning of the input sentence, thus generating less biased and overall better translations. ## results in what follows, first, we show that our model attains bleu scores in the same ballpark as state-ofthe-art approaches such as opus and mbart-50, despite the large gap in terms of parameters or training data. then, we focus our evaluation on the wsd bias, and compare our full-fledged model (baseline+scr) against its baseline variant. ## conclusions in this paper, we presented a fine-tuning strategy that, by leveraging the explicit sense annotations produced by a novel high-precision technique, effectively reduces the disambiguation bias of a baseline neural machine translation model while at the same time also strengthening translation performances, without introducing any requirement at inference time. our analysis on a strong disambiguation system showed that its ability to disambiguate polysemous nouns is worse than that of a baseline nmt model, at least in the studied out-of-domain setting. we believe that this work paves the way for better bias reduction techniques in mt, while also fostering interest in the issue represented by the disambiguation bias. as future work, we plan to further study the ability of nmt models to perform word sense disambiguation and to strengthen research at the intersection of these two fields, with a view to building stronger and more reliable models. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, online. association for computational linguistics.