The AIC System for the WMT 2022 Unsupervised MT and Very Low Resource Supervised MT Task (2022)

Abstract: This paper presents our submissions to WMT 22 shared task in the Unsupervised and Very Low Resource Supervised Machine Translation tasks. The task revolves around translating between German ↔ Upper Sorbian (de ↔ hsb), German ↔ Lower Sorbian (de ↔ dsb) and Upper Sorbian ↔ Lower Sorbian (hsb ↔ dsb) in both unsupervised and supervised manner. For the unsupervised system, we trained an unsupervised phrase-based statistical machine translation (UPBSMT) system on each pair independently. We pretrained a De-Salvic mBART model on the following languages Polish (pl), Czech (cs), German (de), Upper Sorbian (hsb), Lower Sorbian (dsb). We then fine-tuned our mBART on the synthetic parallel data generated by the (UPBSMT) model along with authentic parallel data (de ↔ pl, de ↔ cs). We further fine-tuned our unsupervised system on authentic parallel data (hsb ↔ dsb, de ↔ dsb, de ↔ hsb) to submit our supervised low-resource system.

## introduction like most machine learning approaches, data can be considered the most important component of the recipe for modeling a solution for a given problem. neural machine translation relies heavily on a large amount of training data to correctly model two languages and learn the mapping between them to produce semantically and syntactically right translations. however, machine translation is not available for the majority of the 7000 languages spoken on the earth. this is due to the fact that parallel corpora are scarce or non-existent. there have been several proposals to alleviate the issue of small amounts of parallel data such as pivot translation, multilingual training, and semi-supervised training which resulted in an acceptable performance. unsupervised machine translation became the goto solution when lacking parallel data. the wmt 2022 unsupervised mt task focuses on two very low-resource languages: upper sorbian (hsb) and lower sorbian (dsb). upper and lower sorbian are minority languages spoken in the federal states of saxony and brandenburg in eastern germany. with just 30,000 and 7,000 native speakers, working on these languages is an extreme low-resource task, with little prospect of ever approaching the number of resources available for languages with millions of speakers. however, because they are western slavic languages, the sorbian languages can benefit from czech and polish data @xcite . in this paper, we describe our systems for translating between german ↔ upper sorbian (de ↔ hsb), german ↔ lower sorbian (de ↔ dsb), and upper sorbian ↔ lower sorbian (hsb ↔ dsb) in a both unsupervised and supervised manner. we approach the task by combining two novel approaches for unsupervised machine translation. influenced by @xcite , we start by developing unsupervised phrase-based statistical machine translation systems (upbsmt) for all language pairs independently. in contrast to @xcite , (lample and conneau, 2019) relies on pre-training an xlm model on the source and target language to capture the translation signal instead of using (upbsmt). instead, we benefit from both the pre-training and upbsmt. so, we pre-train an mbart model @xcite on polish, czech, upper sorbian, lower sorbian, and german from scratch as we mentioned earlier that pl and cs are similar to dsb and hsb. we then finetune mbart on synthetic parallel data (de),(de ↔ hsb) and (hsb ↔ dsb) along with authentic parallel data (de ↔ pl, de ↔ cs). we group pl, cs, dsb, hsb under one token slavic while feeding it to the encoder. for our lowresource submission, we fine-tune the unsupervised model on the authentic parallel data provided by the task between (de ↔ hsb),(de ↔ dsb), and (hsb ↔ dsb). our unsupervised approach scored the highest bleu in all directions except (de ↔ dsb) direction. ## related work the earliest approach on unsupervised machine translation was introduced by @xcite where they frame the mt task as a decipherment task, treating the target language as cipher text of english. their method is essentially the same approach taken by cryptanalysts and epigraphers when they use the source texts. they started by estimating the word translation probabilities using a devised iterative em algorithm, due to the huge consumption of memory since they operate on large-scale vocabularies. followed by that, they propose a novel approach based on bayesian decipherment that outperformed the previous em approach in all aspects. after that they build an ngram translation table that was used to estimate an ibm model 3 translation model, the highest bleu @xcite score achieved was 19.3 on the spanish-english opus subtitles data. @xcite provide a two-step solution to unsupervised machine translation. for step one, they start by building an upbsmt system between source and target languages. resulting in two translation models: source-to-target and target-tosource models. using these models, they backtranslate target monolingual data using the targetto-source model to generate ( ŝ rc, trg) pairs that will be used to train the source-to-target neural model. similarly, they back-translate the source monolingual using the source-to-target model to generate (src, t rg) pairs that will be used to train the target-to-source neural model. the second step is training two neural models, source-to-target and target-to-source, using the synthetic data generated from step 1 using iterative back-translation. the first iteration relies solely on data generated by upbsmt, the following iterations substitute a percentage of synthetic data generated by upbsmt by back-translated data from the neural model in the reverse direction. until the whole training data is back-translated from the reverse model. in contrast, (lample and conneau, 2019) starts by pre-training an xlm encoder on masked language modeling (mlm) task on the source and target languages. after pre-training, they initialize an encoderdecoder model using the pre-trained xlm encoder. their training step is composed of three tasks : ## approach inspired by @xcite and (lample and conneau, 2019), we adapted a mixed approach to mitigate the weaknesses and combine the advantages of both methods. @xcite use upbsmt as an explicit initial translation signal to train two translation models from scratch on a translation task. but, upbsmt's output is noisy and the translation model is trained from scratch without any denoising pre-training objective. in contrast, (lample and conneau, 2019) pre-trains an xlm encoder on mlm task and then use it to initialize a seq2seq model which will be trained to translate in an unsupervised manner as we discussed in section 2. although (lample and conneau, 2019) didn't train the translation model from scratch, they relied solely on the three training tasks discussed earlier to capture the cross-lingual translation signal in contrast to @xcite who used an explicit cross-lingual translation signal. we combine the best of both worlds by using upb-smt as our initial translation signal to fine-tune a pre-trained mbart model on a multilingual translation task. ## experiments in this section, we describe our experimental setup and results. readers can refer to our github repository 1 for training scripts, checkpoints, hyperparamters etc. ## results in this section, we present our results on the blind test set of wmt 22 workshop.