Code-Switching with Word Senses for Pretraining in Neural Machine Translation (2023)

Abstract: Lexical ambiguity is a significant and pervasive challenge in Neural Machine Translation (NMT), with many state-of-the-art (SOTA) NMT systems struggling to handle polysemous words (Campolungo et al., 2022). The same holds for the NMT pretraining paradigm of denoising synthetic “code-switched” text (Pan et al., 2021; Iyer et al., 2023), where word senses are ignored in the noising stage – leading to harmful sense biases in the pretraining data that are subsequently inherited by the resulting models. In this work, we introduce Word Sense Pretraining for Neural Machine Translation (WSP-NMT) - an end-to-end approach for pretraining multilingual NMT models leveraging word sense-specific information from Knowledge Bases. Our experiments show significant improvements in overall translation quality. Then, we show the robustness of our approach to scale to various challenging data and resource-scarce scenarios and, finally, report fine-grained accuracy improvements on the DiBiMT disambiguation benchmark. Our studies yield interesting and novel insights into the merits and challenges of integrating word sense information and structured knowledge in multilingual pretraining for NMT.

## introduction lexical ambiguity is a long-standing challenge in machine translation @xcite due to polysemy being one of the most commonly occurring phenomena in natural language. indeed, thanks to a plethora of context-dependent ambiguities (e.g. the word run could mean run a marathon, run a mill, run for elections etc.), words can convey very distant meanings, which may be translated with entirely different words in the target language. to deal with this challenge, traditional statistical machine translation approaches tried to incorporate word sense disambiguation (wsd) systems in mt with mostly positive results @xcite . these were followed by similar efforts to plug sense information in nmt frameworks @xcite . but, since the introduction of the transformer @xcite , the task of disambiguation has largely been left to the attention mechanism @xcite . in the last three years, though, many works have challenged the ability of modern-day nmt systems to accurately translate highly polysemous and/or rare word senses @xcite @xcite . for example, @xcite expose major word sense biases in not just bilingual nmt models like opus @xcite , but also commercial systems like deepl foot_0 and google translate foot_1 , and massively pretrained multilingual models @xcite . a likely explanation is that these models capture inherent data biases during pretraining. this particularly holds for the pretraining paradigm of denoising code-switched text foot_2 -most notably, aligned augmentation (aa, @xcite , where, during the pretraining phase, input sentences are noised by substituting words with their translations from multilingual lexicons, and nmt models are then tasked to reconstruct (or 'denoise') these sentences. aa and subsequent works @xcite @xcite show the benefits of code-switched pretraining for high-and low-resource, supervised and unsupervised translation tasks. despite their success, a major limitation of these substitution mechanisms is that they are unable to handle lexical ambiguity adequately, given their usage of 'senseagnostic' translation lexicons. in fact, in most of these works, substitutions for polysemes are chosen randomly, regardless of context @xcite figure 1 : italian translations of a sentence from the dibimt disambiguation benchmark @xcite by: a) our main baseline, aligned augmentation (aa, @xcite , and b) our approach, wsp-nmt. aa mistranslates the ambiguous word edge as margine @xcite . due to 'sense-pivoted pretraining', wsp-nmt correctly translates it as vantaggio (advantage). @xcite . in an effort to introduce knowledge grounding at the word sense level during pretraining and potentially minimise data errors, enhance convergence, and improve performance, we propose the notion of 'sense-pivoted pretraining' -to move code-switched pretraining from the word level to the sense level. specifically, we propose an approach called word sense pretraining for neural machine translation (wsp-nmt) that first disambiguates word senses in the input sentence, and then code-switches with sense translations for denoisingbased pretraining. figure 1 provides an intuition of how integrating disambiguation in pretraining helps our model handle ambiguous words better, avoiding defaulting to more frequent senses, and reducing errors in translation. indeed, our experiments on using wsp-nmt yield significant gains in multilingual nmt -about +1.2 spbleu and +0.02 comet22 points over comparable aa baselines in high-resourced setups. among other interesting performance trends, we observe that our margin of improvement increases substantially as we move towards low-resource (+3 to +4 spbleu) and medium-resource (+5 sp-bleu) settings. lastly, for more fine-grained evaluation, we also compare our models on the dibimt disambiguation benchmark @xcite for italian and spanish, and note accuracy improvements of up to 15% in the challenging task of verb disambiguation. our key novel contributions are, thus, as follows: multilingual pretraining for nmt the success of multilingual nmt in the latest wmt shared tasks @xcite has heightened the research focus on noising functions used in the denoising-based multilingual pretraining of nmt models. while masking has been successful in scaling to massive models @xcite , there has also been a parallel strand of research exploring more optimal noising functions that grant superior performance at lower data quantities. in particular, code-switched pretraining @xcite @xcite has gained popularity since it moves the denoising task from language modelling to machine translation, and induces superior cross-lingual transfer. notably, pan et al. (2021) proposed aligned augmentation that codeswitches text using muse lexicons, and trained the mrasp2 model -yielding sota results on a wide variety of translation tasks, beating strong baselines pretrained on more data. subsequent research has tried to extend this: li et al. (2022) and reid and artetxe (2022) combine aa-like lexicon-based code-switching with masking, while iyer et al. (2023) and jones et al. (2023) explore optimal code-switching strategies. while effective, none of these works have attempted to move code-switching to the word sense level. most of them sample random sense translations when encountering polysemes, with these errors further propagating during pretraining. li et al. (2022) attempt to partially circumvent this issue by choosing the appropriate sense translation based on the reference sentence, but this trick cannot scale to code-switching (abundant) monolingual data or for code-switching in languages other than the target one. iyer et al. (2023) use pretrained translation and word alignment models to provide empirical gains, but the reliability is low with these black-box systems since such techniques lack grounding. in order to handle lexical ambiguity in a principled manner, we propose wsp-nmt to provide knowledge grounding at the word sense level while pretraining multilingual nmt models. ## conclusion we propose wsp-nmt, a 'sense-pivoted' algorithm leveraging word sense information from kbs for pretraining multilingual nmt models. we observe significant performance gains over the baseline, aa, and note various factors of interest including the impact of wsd quality and morphology. we then study how wsp-nmt scales to various data and resource-constrained scenarios and, finally, note significant accuracy gains in verb disambiguation on the dibimt benchmark. our results help emphasize the utility and strength of wsp-nmt.