E stonian-Centric Machine Translation: Data, Models, and Challenges (2024)

Abstract: Machine translation (MT) research is most typically English-centric. In recent years, massively multilingual translation systems have also been increasingly popular. However, efforts purposefully focused on less-resourced languages are less widespread. In this paper, we focus on MT from and into the Estonian language. First, emphasizing the importance of data availability, we generate and publicly release a back-translation corpus of over 2 billion sentence pairs. Second, using these novel data, we create MT models covering 18 translation directions, all either from or into Estonian. We re-use the encoder of the NLLB multilingual model and train modular decoders separately for each language, surpassing the original NLLB quality. Our resulting MT models largely outperform other open-source MT systems, including previous Estonian-focused efforts, and are released as part of this submission.

## introduction the majority of work on neural machine translation (nmt) is nowadays primarily english-centric, with some notable work on (massively) multilingual mt @xcite @xcite . in recent years, some attention has been directed at translation directions out of english (e.g. this is the primary focus of the wmt'2024 evaluation campaign 1 ) or at pairs that do not include english: for instance, recent wmt and iwslt shared tasks included one or two such pairs @xcite @xcite . in this work, we present our recent efforts on advancing estonian-centric machine translation. in a broader scope the work is part of the neurotõlge project, which develops open machine translation for estonian. 2 the name neurotõlge means neural translation in estonian and the work on its development has started in 2017 and is ongoing. the present contribution covers 18 new translation directions for neurotõlge from and into estonian. we openly release a massive back-translation corpus for these language pairs, extending the synthetic corpus of parallel estonian (synest) @xcite , and release translation models trained using these data. we employ a partially modular approach @xcite in creating translation models. specifically, we use the encoder of an existing massively multilingual translation system nllb @xcite and create the decoders for each target language as separate modules (the architecture is shown in figure 1 ). this setup makes it possible to train the decoders independently, and any subset of the decoders can be deployed afterwards. the achieved translation quality is better than the original nllb system and also surpasses other open systems on the included translation directions. the main contributions of this paper are thus: ## related work in our work, we focus on strengthening the capabilities of open-source mt systems focused on the estonian language. this builds upon previous efforts centered on estonian public translation, most recently, the mtee governmental project @xcite , and, more generally, the neurotõlge project and online translation engine. 2 mtee covered translation between estonian and three other languages: english, german, and russian, and achieved state-of-the-art translation quality at the time @xcite . in this work, we train estonian-centric models for more language pairs, outperforming the mtee models in most cases. instead of training models from scratch, we use the nllb multilingual translation model @xcite as a starting point for our systems. nllb is a massive effort utilizing the multilingual mt approach @xcite , and covering 200 languages, which makes it a convenient base on which to build systems tailored to a smaller number of languages. in this work, we mostly rely on creating large amounts of new training data to improve estonian translation. specifically, we use the backtranslation technique @xcite . existing mt systems are used to generate translations of monolingual corpora into desired languages. the obtained parallel data is then reversed and used to augment the training corpus. thus, the noisy, automatically translated text is on the source side, and the target side contains the cleaner original data, which allows the model to learn text generation based on genuine data. specifically, we use and extend the synest corpus @xcite ), an estonian-focused back-translation dataset, to cover new translation directions and source corpora. in terms of model architecture, our systems are inspired by modular approaches @xcite , where multilingual mt models share encoder and decoder modules for each input and output language instead of having one encoder and one decoder covering all languages. more specifically, we use an existing multilingual encoder module from nllb and train a new decoder for each target language from scratch, somewhat similarly to concurrent work on "mixand-match translation" by @xcite , where encoders and decoders from different models are unified to form a new model. ## extending the synest corpus synthetic corpus of parallel estonian, or synest @xcite ), includes data from the newscrawl monolingual corpus @xcite automatically translated into estonian from 11 languages (arabic, chinese, english, finnish, french, german, latvian, lithuanian, russian, spanish, and ukrainian). the dataset can be used as a back-translated corpus to facilitate training mt models which include estonian. in this work, we significantly extend synest to code target language parallel back-translated corpus total newscrawl paracrawl unpc opensubtitles ## results bleu scores of nllb-1.3b, mtee, madlad-3b, our model, deepl, and google translate on flores-devtest for translation directions from estonian into other languages (our experiments cover german, english, finnish, russian, ukrainian, and chinese as target languages) are shown in from ## deployment and known issues the models are made publicly available on the huggingface model hub 4 and can be run using the tartunlp translation worker. 8 the models are set up in a modular fashion, with one encoder covering all input languages and a separate decoder for each output language. we have found that the models are not robust to some inputs, such as single words; while full the best scores overall are shown in bold, and the best scores among open-source models are underlined. for our model, we report the score of the checkpoint with the best validation loss (the same checkpoint is used for all source languages). with mtee, we show the scores reported by @xcite . ## future work so far the efforts of the project have focused on sentence-level nmt. the next iterations of development and model training will likely focus on document-level mt, either with sequenceto-sequence or decoder-only models. moreover, we are looking into instruction-tuned sequenceto-sequence models: this approach should yield translation-specific emergent abilities and would thus enable the integration of terminologies, onthe-fly domain adaptation, and other types of translation output control. we also plan to dedicate more attention to the robustness of the developed translation engines, for instance, by including upper-cased data in the training dataset for smoother handling of headlines and other all-caps segments, as well as including phrase and word pairs to enhance translation performance when the input is not a complete sentence. ## conclusion in this work, we have made a contribution towards open-source machine translation centered on the estonian language. first, we presented an extended version of the synest synthetic corpus. the new version introduces 12 translation directions from estonian, in addition to previously present directions into estonian. in total, we have generated over 2 billion filtered sentence pairs. we release the full corpus for public use and hope that the availability of this resource will facilitate further work on estonian translation. second, we created new mt models for translation from estonian into 6 languages and from 12 languages into estonian and made them publicly available. evaluation on two benchmarks covering 6 domains has shown that our models are comparable to or outperform previous open efforts on translation from estonian, depending on the language pair and domain, and perform especially well on translation into estonian, outperforming not only previous open-source but also proprietary systems by up to 13 bleu on some domains. these consistent improvements are likely due to the use of massive amounts of synthetic data we created.