GATITOS : Using a New Multilingual Lexicon for Low-resource Machine Translation (2023)

Abstract: Modern machine translation models and language models are able to translate without having been trained on parallel data, greatly expanding the set of languages that they can serve. However, these models still struggle in a variety of predictable ways, a problem that cannot be overcome without at least some trusted bilingual data. This work expands on a cheap and abundant resource to combat this problem: bilingual lexica. We test the efficacy of bilingual lexica in a real-world set-up, on 200-language translation models trained on web-crawled text. We present several findings: (1) using lexical data augmentation, we demonstrate sizable performance gains for unsupervised translation; (2) we compare several families of data augmentation, demonstrating that they yield similar improvements, and can be combined for even greater improvements; (3) we demonstrate the importance of carefully curated lexica over larger, noisier ones, especially with larger models; and (4) we compare the efficacy of multilingual lexicon data versus human-translated parallel data. Based on results from (3), we develop and open-source GATITOS, a high-quality, curated dataset in 168 tail languages, one of the first human-translated resources to cover many of these languages.

## introduction neural machine translation (nmt) has emerged as the dominant way of training machine translation models @xcite , where translation is modeled as a sequence-to-sequence task to be learned by neural networks @xcite . massively multilingual machine translation combination of denoising autoencoding with pretrained cross-lingual embeddings and on-the-fly back-translation to achieve reasonable mt performance with zero parallel data. in our work, we supplement the approach that combines supervised and self-supervised training with multilingual lexica. the motivation for using this resource is as follows. despite the successes of the approach combining supervised and self-supervised training, cross-lingual vocabulary alignment is still highly imperfect in these models, especially for low-resource and unsupervised languages (see @xcite for examples of some common failure modes). that is, training on all languages using a shared set of parameters is insufficient to induce perfect cross-lingual vocabulary alignment. of course, we are not the first to experiment with multilingual lexica to improve nmt performance, or multilingual nlp applications more generally; section 2 gives more details. using the publicly available massively multilingual lexicon panlex @xcite , we demonstrate that this added lexical data leads to small but significant gains over a baseline model on average, even for high-resource languages; and with smaller but carefully curated bilingual lexica, the gains are substantially larger. in both cases, the gains are most significant for unsupervised and low-resource languages. our contributions are as follows: the tl;dr of this paper is that bilingual lexica help low-resource and zero-shot nmt in almost all cases, and that most training-time augmentation methods have similar efficacy, and can be combined to be more effective. when scaling up to larger and more expressive models, these methods retain their efficacy, but the quality of the translated bilingual lexica becomes more important than the sheer quantity of lexical data points used for data augmentation. for instance, small, high-quality lexica like gatitos show about 5x larger chrf improvement than larger, noisier lexica like panlex. throughout this paper, experiments are done on only 24 gatitos languages; based on the success of these, we expand the dataset to 170 languages, all low-resource, and open-source it. ## related work a number of works have looked at using multilingual lexicon data augmentation for nmt and other nlp tasks. the first class of augmentations that we experiment with is "codeswitching," where words in the source sentence are swapped out for their dictionary translations to create mixedlanguage sentences. this approach has been used for a range of multilingual nlp tasks, including @xcite @xcite @xcite @xcite @xcite @xcite . many of these, however, only look at codeswitching between the source and target languages, e.g. substituting source words with dictionary translations into the target language, or word-for-word bilex translations of the target to make synthetic back-translated data @xcite . @xcite experiment with codeswitching on nli, sentiment classification, document classification, dialogue state tracking, and spoken language understanding, malon (2021) looks at codeswitching embeddings for language modeling, and @xcite experiment on ner, pos tagging, and dependency parsing. another similar work is @xcite , in which the mlm task is modified such that instead of predicting masked source tokens in the source language, the authors provided language embeddings to cue the model to predict the masked tokens in a different language instead. codeswitching augmentations go by a variety of different names, e.g. "dictionary denoising" (reid and artetxe, 2022), "random aligned substitution" @xcite , or "code-mixing". in our paper, we will stick to the term "codeswitching," though we will try to point out where an identical or similar approach has been tried under a different name. the second class of augmentations we experiment with involves prepending lexicon translations to source sentences as additional cross-lingual signal, as instead of swapping out words in the source sentence. this approach has been tried as well for enhancing mt performance, e.g. in @xcite ; @xcite ; niehues (2021); @xcite ; zhong and chiang (2020); @xcite , and for similar tasks like language modeling @xcite . one potential advantage this approach has over the codeswitching method is that it can be applied at inference time as well: multilingual lexicon entries can be prepended to sentence queries to steer the model toward more accurate word translations. outside of nmt models, this lexical prompting approach has also been applied to translation with llms: @xcite provide llms with dictionary translations of some of the source sentence words, which the model can use to cover gaps in its vocabulary coverage (although the authors do not experiment with truly low-resource languages). with the rise in popularity of llms for mt and other tasks, this is an exciting area for further research. ## training data models are trained on sentence-level web text. ## evaluation we use two translation evaluation sets: @xcite @xcite , an open-sourced evaluation set consisting of 2009 english wikipedia sentences translated by humans into 200 languages, and gatones, an in-house evaluation set of 1,200 english sentences translated into various languages. we use the sacrebleu (post, 2018) implementation of chrf foot_2 for our evaluation metric. higherquality, embedding-based metrics like bleurt @xcite are not available for these languages, and chrf seems to be the one of best of the surface-level metrics for low-resource languages @xcite @xcite . for this study, we only evaluate on englishcentric directions. the reason is that, although both evaluation sets are multi-way parallel, they are both also english-original, and thus lose the same infor-mation that pivot translations do. furthermore, this study places more weight on the @xmath0. the main reason for this is that the @xmath1, so the @xmath2; as a result we care more about improving this direction. ## methods in this paper, we divide our augmentation approaches into two classes: "codeswitching" approaches, which involve substituting source sentence words for their dictionary translations, and "glowup," which entails prepending dictionary translations of source words to source sentences. the main difference between these approaches is whether dictionary translations are substituted for source text (in the case of codeswitching) or added to the sentence (in the case of glowup). as a third augmentation, we experiment with training on raw lexicon token pairs directly, treating them like any other parallel data. the novelty of our contribution lies not so much in any one of our methods, but rather in (1) the application of these methods to unsupervised machine translation; (2) the number of methods we compare in controlled experiments; (3) the scale of our experiments, in terms of number of languages, data quantity, and model capacity; and (4) the application of these methods to "in the wild" web-crawled data. while a variety of papers (e.g. reid and artetxe (2022); yang et al. ( 2020 )) have explored specific augmentations on particular language pairs, we believe our paper is the first to undertake a rigorous comparison of different augmentation strategies across hundreds of languages in a real-world setting. ## results we evaluate all our models on the flores-200 dataset @xcite @xcite , which contains englishaligned parallel sentences for 200 languages. in appendix section b we also report scores on the in-house gatones eval set. we use the following resourcedness classifications for our analysis: despite these trends seeming robust across models, the effect sizes are relatively small, maxing out at about +1.5 average ∆chrf gain. however, the picture changes dramatically when only looking at the subset of languages that has the higher-quality gatitos training lexica. for these 26 languages, every augmentation, even the parallel ones, have large performance gains. the win-ning augmentations for urls remain codeswitch-monogatipanlex and glowupmonogatipanlex, the former having an average gain of +7.0chrf on flores-200and +8.0chrf on gatones, and the latter having +5.6chrf on flores-200 and +9.5chrf on gatones. finally, although it was not the first place model in any category, the gatipanlextokenpairs has large gains in all directions over baseline, and only falls short of the more complex augmentations by a small margin. furthermore, when used in conjunction with either glowup or codeswitch, it further improves performance across all categories. this may be one of the most useful long-term findings of this study: raw token pairs perform roughly on par with all the other fancier augmentations! 8.1 scaling up: bigger models, more data sometimes, results on smaller models do not transfer to larger models. to this end, we train larger transformer models with 1.6b parameters using 10× the parallel data and 10× the high-resource monolingual data. we trained three large models: a baseline, a token-pair model, and a token-pairs + codeswitch-mono model. the chrf scores for these models can be seen in is this the bitter lesson (sutton, 2019) getting us again? perhaps-but the picture is less bleak than it first appears. when we look at the subset of the languages where we have a higher assurance that the bilingual lexica are higher-quality-namely, those that use gatitos bilingual lexica-we still see consistent wins.for these 26 languages, all models see consistent gains, and the gains are biggest on unsupervised languages. overall, the takeaway from these experiments is that one has to ensure that the data is of high quality when applying lexical data augmentation at such a large scale. while we saw substantial improvements for many languages, these were balanced out by losses for other languages (especially those with only panlex, but not gatitos, data). ## conclusions in this paper we explore the ways that that augmenting training data with bilingual lexicon information can improve the performance of machine translation models on low-resource and unsupervised languages, and open-source the gatitos dataset, which leads to average gains of about +7 chrf on unsupervised languages. we perform extensive experimentation with three main types of lexical augmentation: codeswitching, lexical prompting, and raw token-pair training. the results show that applying any of these augmentations to monolingual data yields substantial improvements, and that they can be combined for greater effect. the leader (by a small margin) is the combination of codeswitchmono and raw token-pair training. these results hold when scaling up model and data size, but in the settings with more data and larger models, the quality of the bilingual lexica plays a relatively bigger role, and augmentation with the noisier panlex begins to lag in quality behind the much smaller, yet higher-quality, gatitos dataset. future work will likely want to focus on prompting foundation models with bilingual lexica. large language models show promise on machine translation for high-resource languages @xcite , but their capabilities on low-resource languages have yet to be thoroughly explored. additionally, a more thorough investigation of the trade-off between cost and quality for tiny datasets can be explored: with a fixed budget of time or money, should one spend their time translating text, making monolingual text, or making bilingual lexica?