D i B i MT : A Novel Benchmark for Measuring W ord S ense D isambiguation Biases in M achine T ranslation (2022)

Abstract: Lexical ambiguity poses one of the greatest challenges in the field of Machine Translation. Over the last few decades, multiple efforts have been undertaken to investigate incorrect translations caused by the polysemous nature of words. Within this body of research, some studies have posited that models pick up semantic biases existing in the training data, thus producing translation errors. In this paper, we present DiBiMT, the first entirely manually-curated evaluation benchmark which enables an extensive study of semantic biases in Machine Translation of nominal and verbal words in five different language combinations, namely, English and one or other of the following languages: Chinese, German, Italian, Russian and Spanish. Furthermore, we test state-of-the-art Machine Translation systems, both commercial and non-commercial ones, against our new test bed and provide a thorough statistical and linguistic analysis of the results. We release DiBiMT at https://nlp.uniroma1.it/dibimt as a closed benchmark with a public leaderboard.

## introduction the polysemous nature of words poses a longstanding challenge in a wide range of natural language processing (nlp) tasks such as word sense disambiguation (navigli, 2009; bevilacqua et al., 2021) (wsd), information retrieval (krovetz and croft, 1992) (ir) and machine translation (emelin et al., 2020) (mt). in mt, some research works have addressed the ability of systems to disambiguate polysemous words. for instance, given the sentence he poured a shot of whiskey, the polysemous target word shot unequivocally means a small quantity and therefore a possible translation into italian could be: versò un goccio di whiskey. however, some mt systems propose the following translation: versò uno sparo in the last few decades, attempts have been made to investigate the aforementioned phenomena. in fact, recent studies have observed a direct correlation between semantic biases in the training data and semantic errors in translation. however, their findings are limited by the following shortcomings: i) they are not based on entirely manually-curated benchmarks; ii) they rely heavily on automaticallygenerated resources to determine the correctness of a translation; and iii) they do not cover multiple language combinations. in this work, we address the aforementioned drawbacks and present dibimt, to the best of our knowledge the first fully manually-curated evaluation benchmark aimed at investigating the impact of semantic biases in mt in five language combinations, covering both nouns and verbs. this benchmark allows the community not only to better explore the described phenomena, but also to devise innovative mt systems which better deal with lexical ambiguity. specifically, the contributions of the present work are threefold: he poured a shot of whiskey. target word is shot, in its meaning of a "small drink of liquor". we expect translations to contain, for example in italian, goccio (lit. a drop), but not, for example in spanish, pistolero (a person who shoots). to enable further research, we release dibimt as a closed benchmark with a public leaderboard at https://nlp.uniroma1.it/dibimt . ## related work over the course of the last few decades, several approaches to the evaluation of the lexical choice in mt have been proposed. to this end, cross-lingual benchmarks were created in which systems were required to provide the translation or a substitute for a given target word in context in a target language (vickrey et al., 2005; mihalcea et al., 2010; lefever and hoste, 2013). more recently, gonzales et al. (2017) put forward contrawsd, a dataset which includes 7,200 instances of lexical ambiguity for @xmath0, and 6,700 for @xmath1. this dataset pairs every reference translation with a set of contrastive examples which contain incorrect translations of a polysemous target word. for each instance, the answer provided by systems is considered correct if the reference translation is scored higher. based on a denoised version of the contrawsd dataset and focusing on the language combination @xmath2, gonzales et al. (2018) present the word sense disambiguation test suite which, unlike contrawsd, evaluates mt output directly rather than by scoring translations. the suite consists of a collection of 3,249 sentence pairs in which the german source sentences contain one ambiguous target word. as target words, the authors considered only words in german whose translation into english does not cover multiple senses, thus making the evaluation more straightforward. despite their effectiveness, such benchmarks do not allow systems to be tested in multiple language combinations, and only cover a very limited number of words and senses. to address these limitations, raganato et al. (2019) proposed mucow, an automatically-created test suite covering 16 language pairs, with more than 200,000 sentence pairs derived from word-aligned parallel corpora. other research studies investigated the disambiguation capabilities of mt systems by exploring their internal representations (marvin and koehn, 2018; michel et al., 2019), or improving them via context-aware word embeddings (liu et al., 2018). more recently, emelin et al. (2020) introduced a statistical method for the identification of disambiguation errors in neural mt (nmt) and demonstrated that models capture data biases within the training corpora, which leads these models to produce incorrect translations. although the authors expected their approach to be transferable to other language combinations, they only focused on @xmath3. based on the findings and open research questions raised in the aforementioned works, the present paper aims at investigating not only the presence, but also, most importantly, the nature and properties of semantic biases in mt in multiple language combinations, via a novel entirely manually-curated benchmark called dibimt and a thorough performance analysis. ## building dibimt the dibimt benchmark focuses on detecting word sense disambiguation biases in nmt, i.e., biases of certain words towards some of their more frequent meanings. the creation of such a dataset requires i) a set of unambiguous and grammaticallycorrect sentences containing a polysemous target word; ii) a set of correct and incorrect translations of each target word into the languages to be covered. figure 1 depicts an example of a dataset item. ## results and discussion we now: i) use dibimt to carry out an evaluation of 7 different machine translation systems; ii) report the obtained results, including a thorough statistical and linguistic evaluation; iii) extensively discuss our findings, providing multiple measures @xcite a more detailed description of the analysis procedure is provided in appendix a. ## conclusions in this work, we presented dibimt, a novel benchmark for measuring and understanding semantic biases in nmt, which goes beyond simple accuracy and provides novel metrics that summarize how biased nmt models are. we tested dibimt on 7 widely adopted nmt systems, extensively discussing their performances and providing novel insights into the possible causes and relations of semantic biases within nmt models. furthermore, statistics of our annotations suggest that, when dealing with translations, synsets' lexicalizations cannot be used interchangeably, as their choice depends heavily on the context. in the future, we plan to improve dibimt by introducing better heuristics to recognize and handle miss cases, especially covering the linguistic phenomena we described (see section 4.2); we also aim at widening language coverage and increasing the number of sentences in the benchmark, consequently improving word and sense coverage. to enable further research, we release dibimt as a closed benchmark with a public leaderboard at: https://nlp.uniroma1.it/dibimt . @xcite we do this to make the translations more grammatically fluent, and not to correct the disambiguation of the target term, which was never detected as being wrong in the sampled cases.