Improving Machine Translation of Rare and Unseen Word Senses (2021)

Abstract: The performance of NMT systems has improved drastically in the past few years but the translation of multi-sense words still poses a challenge. Since word senses are not represented uniformly in the parallel corpora used for training, there is an excessive use of the most frequent sense in MT output. In this work, we propose CmBT (Contextually-mined Back-Translation), an approach for improving multi-sense word translation leveraging pre-trained cross-lingual contextual word representations (CCWRs). Because of their contextual sensitivity and their large pre-training data, CCWRs can easily capture word senses that are missing or very rare in parallel corpora used to train MT. Specifically, CmBT applies bilingual lexicon induction on CCWRs to mine sense-specific target sentences from a monolingual dataset, and then back-translates these sentences to generate a pseudo parallel corpus as additional training data for an MT system. We test the translation quality of ambiguous words on the MuCoW test suite, which was built to test the word sense disambiguation effectiveness of MT systems. We show that our system improves on the translation of difficult unseen and low frequency word senses.

## introduction recent nmt systems have remarkable performance for many languages @xcite but there are still numerous areas for improvement. one such important area concerns the disambiguation and translation of multisense words. it is particularly challenging to mt systems as sense distribution is skewed with some senses rarely seen or missing in the parallel corpora. this results in the mt system producing translation errors for these rare/unseen senses, causing incomprehensible output sentences. in this work we aim at improving the translation of rare and unseen senses of ambiguous words. previously @xcite showed that encoder-decoder based nmt systems integrate information relevant for wsd into the encoder hidden states, but finding the correct sense is still a challenging task and nmt systems are biased toward the most frequent senses of words @xcite . disambiguation errors are often due to over-reliance on training data artifacts, such as frequent word co-occurrences (e.g. hot spring is always translated as the thermal activity and not as a season), instead of a deeper understanding of the multi-sense words given the input sentences @xcite . additionally, mt systems tend to learn and use frequent words more often and disregard less frequent ones @xcite . previous work has improved the translation of ambiguous words, e.g., by leveraging lexical resources @xcite or sense specific embeddings @xcite , but they are restricted to the senses seen in the parallel training corpus and not trained on missing senses. in contrast, we propose a method to mine addi-tional data containing the contextual translations of rare and unseen senses without relying on them being in parallel corpora. our method, called cmbt (contextually-mined back-translation), relies on contextualized cross-lingual word representations (ccwrs) to translate source language multi-sense words and find target language sentences containing the translations of the right senses. we then build a synthetic parallel corpus by back-translating these sentences, making sure that the original multisense words are contained on the source side, in order to extend the training corpus for better sense coverage. we illustrate our method in figure 1 . the advantage of our approach is that ccwrs, such as mbert @xcite or xlm-r @xcite , can be trained on cheap and large monolingual corpora covering a wide frequency range of word senses. by leveraging ccwr-mined sentences containing the translations of these senses in the form of a synthetic parallel corpus, our mt system is not restricted to frequent senses seen in the parallel training corpus. we test our approach on @xmath0. although it was built to test overall wsd performance of mt systems, we create subsets of the provided training corpus to test on unseen and rare senses more directly. our experiments show that using mined sentences as additional data for our nmt systems consistently improves the translation performance (f 1 ) of rare and unseen senses. our proposed approach can be effectively applied to other language pairs as well, since the required resources are widely available. ## related work the problem of wsd is long-standing and extensively studied. multiple neural systems were proposed, e.g., by relying on sequence-to-sequence architectures @xcite , using sense embeddings @xcite or pre-trained language models @xcite . it was shown that wsd positively impacts the performance of downstream applications, such as information retrieval @xcite , sentiment analysis @xcite or topic classification @xcite . wsd is an important problem for mt as well. previously it was shown that the translation performance can be improved by integrating word sense information into mt systems. in @xcite sense labels were assigned to each multisense word using k-means clustering, which served as additional information for a statistical mt system. similarly, explicit word sense information using wordnet was integrated into nmt systems in @xcite . @xcite leveraged sense embeddings induced by specialized lstm modules, while lexical chains of semantically similar words within a document were employed in @xcite . although these approaches do improve wsd performance, they are restricted to the senses seen frequently in the parallel training corpus. in contrast, we focus on the improvement of senses that are missing or very rare by building a synthetic parallel corpus containing these senses using back-translation @xcite . similarly, @xcite back-translated a carefully selected set of sentences to improve the translation of out-of-vocabulary (oov) words, i.e., words that are contained in the text to be translated but not in the training corpus. they used bilingual fast-text @xcite embeddings to find all translations of oovs independent of their contexts. in contrast, we consider the whole sentence when translating multi-sense words in order to determine the right translation of the right sense used in the right context using ccwrs, which we show to be crucial to improve missing and rare sense translation. @xcite proposed a data augmentation approach to adapt mt systems to novel vocabulary in human-submitted translations using ccwrs. they generate training samples for the novel words by mining parallel sentence pairs with similar contexts and adding the novel words to them. in contrast, our approach does not rely on parallel sentences, using only monolingual data and back-translation. various datasets were proposed to test wsd, such as those released by the series of senseval @xcite and semeval @xcite @xcite shared tasks. while most datasets are monolingual, @xcite introduced xl-wsd supporting 18 languages allowing to evaluate zero-shot cross-lingual wsd approaches. to test how well mt systems can disambiguate multi-sense words in their outputs @xcite created the parallel corpus called con-trawsd where for each source sentence containing an ambiguous word two translations are given with the correct and incorrect senses respectively which have to be scored by the mt systems. the mucow dataset was introduced for a more direct evaluation where instead of scoring given target language sentences the translations of multi-sense words in the mt systems' outputs are evaluated @xcite . we use mucow to evaluate our approach. ## approach the goal of cmbt is to incorporate contextdependent word translation that is able to deal with rare and unseen senses and leverage cheap monolingual data as additional training data for our nmt system for better multi-sense word translation. the main steps of our approach are the following: i) we detect multi-sense words in the source side of test corpus using babelnet @xcite , which ii) we translate using ccwrs and mine target language sentences containing these translations. iii) we back-translate these sentences to the source language using a baseline nmt system. we use a special marker placed in the target language sentences, which are replaced with the multi-sense words on the source side, in order to ensure the presence of rare and unseen senses in the new corpus. finally, we fine-tune our base nmt system using the gold and additional synthetic parallel data. we summarize the pipeline in our approach in figure 1 and detail the three main steps below: ## results and discussion unseen and rare sense translation we present both the intrinsic performance of bwes @xcite or cmbt (xlm-r) based word translation (acc@n) and extrinsic mt system based translation (f 1 ) in in terms of acc@n cmbt word translation performs significantly better than bwes. this is not surprising, since the context independent bwes predict the same translations for a given multisense word for each sentence it is contained in. in contrast, xlm-r shows a better wsd performance by relying on the context in the sentences. our improved nmt system using cmbt outperforms the baseline system in all setups in terms of f 1 . it is especially effective on the unseen and very rare senses due to the additional synthetic sentence pairs containing these senses and their translations. in addition, it is also effective for the rare senses in the higher relative frequency range bin. bwes based mining is also helpful for the unseen and very rare senses but it is less effective compared to cmbt. although bwes are context independent, by taking top-5 translations some of the senses can still be improved. on the other hand, bwes have minor negative effects for the higher frequency range. ## conclusions in this paper we proposed cmbt, a simple and effective approach for improved rare and unseen word sense translation. it serves as a general framework that effectively exploits the contextdependent cross-lingual correspondence from a pre-trained ccwr for an mt system. we show cmbt brings significant improvements for multisense word translation on the @xmath1. the improvements are the most pronounced when we directly targeted the evaluation of the difficult rare and unseen senses. as the only requirement of cmbt, on top of the parallel data necessary for the training of the mt system, is a monolingual corpus and an off-the-shelf pretrained multilingual model, cmbt can be applied 1. src for example, a wife learns that her husband put money in the bank in his name rather than in a joint account. top-1 der abfluss bei einer überweisung erfolgt im zeitpunkt der abgabe des überweisungsauftrags an die bank... top-1 bt the flow of a transfer is made when the contract is delivered to the bank... top-2 osama und yeslam bin laden hatten von 1990 bis 1997 ein gemeinsames konto bei der schweizer bank ubs. top-2 bt osama and yeslam bin laden shared an account at the swiss bank ubs between 1990 and 1997. 2. src at this decisive moment in dutch history my father was positioned on the bank of the river waal near the city of nijmegen. top-1 der highway führt nördlich am stadtzentrum vorbei und gelangt von dort an das ufer des ontariosees. top-1 bt the highway passes north of the center of the city and then reaches the bank of lake ontario.