Improving Translation of Out Of Vocabulary Words using Bilingual Lexicon Induction in Low-Resource Machine Translation (2022)

Abstract: Dictionary-based data augmentation techniques have been used in the field of domain adaptation to learn words that do not appear in the parallel training data of a machine translation model. These techniques strive to learn correct translations of these words by generating a synthetic corpus from in-domain monolingual data utilising a dictionary obtained from bilingual lexicon induction. This paper applies these techniques to low resource machine translation, where there is often a shift in distribution of content between the parallel data and any monolingual data. English-Pashto machine learning systems are trained using a novel approach that introduces monolingual data to existing joint learning techniques for bilingual word embeddings, combined with word-for-word back-translation to improve the translation of words that do not or rarely appear in the parallel training data. Improvements are made both in terms of BLEU, chrF and word translation accuracy for an En->Ps model, compared to a baseline and when combined with back-translation.

## introduction one difficulty of low-resource neural-machine translation (nmt) is the ability of models to correctly predict words that are out of vocabulary (oov). oov words are of particular interest when working with low-resource language pairs as such pairs generally exhibit a more significant shift in the distribution of content between the training and test data compared to high-resource language pairs. the available training data for low-resource languages often contains a significant amount of content from specific domains such as it and religious texts @xcite which is not the case in common down-stream tasks for nmt systems such as translating news articles. hence, the task of improving the prediction of oov words in low-resource nmt has significant benefits when deploying such models in realistic inference scenarios. additionally, the overall low amount of parallel data inherent in the task means that the vocabulary covered by the training data is naturally smaller than the vocabulary covered in a more well-resourced nmt scenario. this work aims to improve the prediction of target side oov words for an english-pashto (en-ps) nmt system. to improve the translation of oov words we incorporate monolingual target-side data when training nmt models by generating synthetic source-side sentences. incorporating monolingual target-side data using back-translation (bt) @xcite has been shown to improve the overall performance of low-resource nmt systems in proceedings of the 15th biennial conference of the association for machine translation in the americas orlando, usa, september 12-16, 2022. volume 1: research track terms of automatic sentence-level evaluation metrics such as bleu or chrf. however, an important benefit of incorporating monolingual data is the increase in the amount of vocabulary that is observed during training, the effects of which can only be seen when evaluating nmt predictions at the level of single oov words. work in the field of domain adaptation has shown that word-for-word (wfw) back-translation using bilingual dictionaries extracted from bilingual word embeddings (bwe) is a suitable alternative to bt when specifically targeting improved translation of oov words @xcite . whilst the source-side sentences produced by wfw-bt have lower adequacy and fluency, the key benefit compared to bt is that they more frequently result in direct supervision of oov words. that is to say, wfw-bt more frequently results in source-side sentences that contain a correct translation of target side oov words and, by extension, improve the ability of the nmt model to predict those words correctly. inspired @xcite we adopt the wfw-bt methodology to improve the prediction accuracy of oov target side words for the en-ps nmt model. compared to the dictionary-based techniques in domain adaptation @xcite , which aim to predict target-side words specific to the domain correctly, our goal is to correctly predict items from the more varied monolingual vocabulary which are not present in the restricted parallel vocabulary. the consequence of this is that the task is not targeted at a specific set of vocabulary, and by extension, examples of oov words are less frequent. additionally, oov words are less likely to appear in similar contexts on both sides of the monolingual data because we rely on the assumption that the distribution of content is the same across languages. this assumption only holds weakly for english and pashto, which are both linguistically and culturally distinct @xcite . moreover, the morphological complexity of pashto means that many words have considerably more surface forms than their english counterparts, all of which should all translate to the same english word. as a result of the above observations obtaining a bilingual embedding space (bwe) that correctly maps not only frequent words but specifically oov words is challenging. we propose a new approach to obtaining bwes based on the findings of @xcite ; @xcite that joint training @xcite leads to more isomorphic bwe spaces for linguistically distinct languages. however, joint training requires parallel data to train and hence only maps the embedding spaces of words in the parallel data. our approach trains on the parallel data using joint training to anchor an embedding space whilst simultaneously training on monolingual data. in addition, we incorporate sub-word information into the joint trai -'v bm,ning approach as we hypothesise that sub-word information will help alleviate the data sparsity due to pashto's morphological complexity. the main contributions of this work are as follows: hu et al. (2019); huck et al. (2019); peng et al. (2020) all use dictionary-based methods for data augmentation in a domain adaptation setting focusing on oov words. hu et al. (2019); huck et al. (2019) both use bilingual lexicon induction (bli), but do so in a high resource setting with artificial monolingual data, which is generated by selecting alternating sentences from a parallel corpus as monolingual data. peng et al. ( 2020 ) make use of a high quality pre-existing dictionary to learn new translations. our work also uses bli and wfw-bt; however, we apply these methods to a genuine low-resource nmt problem. rather than learning the translations of a specialised subset of vocabulary from monolingual data that contains these words on both sides, we are trying to train nmt models to correctly predict words outside the more specialised vocabularies often found in low-resource parallel data sets. wfw translations also play an important role in unsupervised nmt (unmt), where they are used to bootstrap nmt models @xcite before applying iterative bt. however, unmt requires careful model choices, works poorly when languages have low amounts of monolingual data @xcite and neglects the fact that there is often a small amount of parallel data available. additionally, it does not focus on the correct prediction of oov words but rather on the sentence-level translation quality. this work directly uses the available parallel data to train nmt models and uses the monolingual data to improve the prediction of oov words. mapping-based approaches have been the dominant methodology for bli, reporting strong results whilst only requiring weak supervision or no supervision by using discriminator networks or identical tokens @xcite . these methods are based on the assumption of isomorphism between word embedding spaces @xcite and require sufficient monolingual data to learn semantically meaningful word embeddings @xcite . @xcite propose joint training of bwe spaces using automatically extracted word alignments as a parallel signal and @xcite observe that joint training leads to increased isomorphism. @xcite introduce an anchor-based method to improve bli from low-resource language pairs. this work builds on @xcite by incorporating monolingual data and sub-word information to learn stronger bwe spaces. unlike other bli tasks, which are often only evaluated on words that appear relatively frequently, we are specifically interested in the bli performance on less frequent words. @xcite proposed mbart, a masked language model (mlm) sequence-to-sequence pre-training that aligns the token level representations across many languages. along with bt @xcite mlm pre-training is the most common way of incorporating monolingual data. we incorporate a mbart-like foot_1 methodology when training our nmt models to ensure a strong baseline. @xcite find that for low-resource languages static word embeddings perform better than mlm on bli tasks which they attribute to a better lexical alignment. based on this @xcite combine mlm with bwes to initialise unmt models. whilst the aim of our work is different, these results demonstrate that bwes are still a suitable tool for learning alignments between lexical items and, by extension, improving the prediction of oov words. finally, mrasp @xcite is an alternative to mlm whereby words and phrases are brought into a similar representation space by substituting aligned words in parallel data sets using dictionaries. mrasp is more closely linked to our work than mbart as it focuses on introducing aligned words during pre-training. ## methodology our approach is split into two distinct stages. the first is obtaining a pseudo-parallel corpus, and the second is training nmt models using the corpus. below we outline the approaches used to obtain a bwe space by combining joint training with monolingual data, extracting a dictionary from the bwes and how the dictionary is used to translate target-side monolingual data. together these three steps represent the wfw-bt methodology which is used to obtain the pseudo-parallel corpus. when learning the bilingual embedding spaces, sub-word information is incorporated either by using fasttext @xcite for mapping-based approaches or by representing words as a combination of n-grams in the same manner as fast-text for bivec approaches. ## experimental design the experimental setup is chosen to investigate a genuine low-resource language paired with english. ## results ## conclusion the bwe evaluation results show that muse correctly translates more oov words than the proposed combined bivec approach, where more weight is given to sentence-level accuracy results as they cover a higher proportion of oov words. however, when viewed in the context of nmt, it appears that the wfw back-translations using bivec lead to more oov and rare words being correctly predicted. for oov words we hypothesise that this is due to the wfw-bt model being able to leverage the higher overall quality of the bivec combined backtranslations to predict more oov words correctly. specifically, this means that bivec combined results in a higher proportion of context words of the oov word being translated correctly. regarding nmt, the results demonstrate that incorporating word-level translations benefits the model even when using back-translation when the low-resource language is on the target side. however, the results are less conclusive when the high-resource language is the target language. the low recall for all oov words for the nmt task suggests that even when the dictionary contains accurate translations, it is difficult for these to transfer into correct model predictions. as a result, it seems that incorporating word-level translations from the monolingual data can benefit the model. it may be that for languages that exhibit a different sentence structure, wfw back-translation is not the best methodology for incorporating the oovs and rare words. instead, an approach of inserting them into back-translations or existing parallel data may be more appropriate to ensure a higher degree of fluency in the synthetic sentences.