Findings of the WMT 2022 Shared Tasks in Unsupervised MT and Very Low Resource Supervised MT (2022)

Abstract: We present the findings of the WMT2022Shared Tasks in Unsupervised MT and VeryLow Resource Supervised MT with experiments on the language pairs German to/fromUpper Sorbian, German to/from Lower Sorbian and Lower Sorbian to/from Upper Sorbian. Upper and Lower Sorbian are minoritylanguages spoken in the Eastern parts of Germany. There are active language communitiesworking on the preservation of the languageswho also made the data used in this Shared Taskavailable.In total, four teams participated on this SharedTask, with submissions from three teams for theunsupervised sub task, and submissions fromall four teams for the supervised sub task. Inthis overview paper, we present and discuss theresults.

## introduction for a large majority of the world's languages, only limited resources are available to train and provide nlp tools. the need for parallel data in a (supervised) translation scenario aggravates this problem further. the shared tasks in unsupervised mt and very low resource supervised mt aim at promoting the research on translating low and very low resourced languages. following the shared tasks in the two previous years @xcite , we continue to cooperate with the sorbian community, namely the sorbian institute 1 and the witaj sprachzentrum (witaj language center) 2 for this year's shared task. we offer all translation directions between the languages upper sorbian, lower sorbian and german, for both supervised and unsupervised translation. upper and lower sorbian are minority languages spoken in the eastern part of germany in the 1 https://www.serbski-institut.de/en/institute/ 2 https://www.witaj-sprachzentrum.de/ federal states of saxony and brandenburg. with only 30k and 7k native speakers, there are only few resources available. however, as western slavic languages, upper and lower sorbian are closely related to polish and czech and can thus make use of the comparatively large data sets available for those languages. in this year, four teams participated in the shared task, resulting in three to four submissions for each language pair for both the supervised and unsupervised variants. ## tasks and evaluation in contrast to the previous shared tasks, all language combinations between upper sorbian, lower sorbian and german are considered, resulting in the six following translation pairs: factoring in the variants supervised and unsupervised translation for each language pair, there is a total of 12 translation pairs. for the evaluation, we follow the strategy employed in the previous shared task and use bleu scores @xcite and chrf scores @xcite as implemented in sacrebleu @xcite . 3 furthermore, we evaluate the submissions using bertscore (zhang et al., 2020) 4 with xlm-roberta large @xcite as an underlying model for translations into german foot_2 . we decided against using comet scores @xcite . this metric considers both the source language and the target language, but because it relies on xlm-r models, it does not support the sorbian languages. ## data to allow for a direct comparison between the different submissions, we only allowed training based on the resources released for the task, as well as resources for related languages (german, czech and polish data from the wmt news tasks foot_3 or available in the opus project foot_4 ). in particular, the use of large models pre-trained on large data sets was not allowed. for the unsupervised translation sub-task, we restricted the the data set as follows: all released upper/lower sorbian data could be used, with the exception of the parallel upper sorbian ↔ lower sorbian corpus. furthermore, the german side of the parallel german ↔ upper sorbian and german ↔ lower sorbian training corpora was excluded. this setup allowed us to make maximum use of the low-resourced languages without providing parallel data. ## submitted systems four teams participated in the supervised sub-task foot_5 , and three teams participated in the unsupervised sub-task. we present a brief system description of each team's submission, with an overview of the results listed in tables 2 to 7. aic @xcite ) for the unsupervised system, they trained an unsupervised phrase-based statistical machine translation (upbsmt) system on each pair independently. they trained a de-slavic mbart model from scratch (random initialization) on the following languages: polish (pl), czech (cs), german (de), upper sorbian (hsb), and lower sorbian (dsb). they then fine-tuned their mbart on the synthetic parallel data generated by the upbsmt model along with authentic parallel data (de ↔ pl, de ↔ cs). they further fine-tuned their unsupervised system on authentic parallel data (hsb ↔ dsb, de ↔ dsb, de ↔ hsb) to submit the supervised low-resource system. muni nlp @xcite this team submitted supervised nmt systems for the lower sorbian-german and lower sorbian-upper sorbian language pairs, in both translation directions. they employed a new subword tokenization algorithm, high frequency tokenizer (hft), to obtain more meaningful subword vocabularies. they tested this against bpe in the first round of experiments where they trained two different models on the data tokenized with each tokenizer, so four systems in total: two standard transformers and two transformers with hyperparameters optimized for the dataset size. they then followed the data diversification procedure @xcite generating and collating authentic and synthetic data alternatively from each previous system and the original parallel data to create an augmented dataset. then, they trained a transformer model on these new data, tokenized with hft, to obtain the final system. thus, the approach is based only on the original parallel corpus. huawei tsc @xcite huawei translation services center participated in all 6 supervised tracks. their systems are build on deep transformer models with a large filter size. first, they selected a base multilingual model with german-czech (de-cs) and german-polish (de-pl) parallel data for all of the 6 tracks. they then utilized regularized dropout (r-drop), back translation, fine-tuning and ensemble multilingual models to improve on the best individual system performance. for the unsupervised task submission, they applied their pre-trained multilingual system with zero-shot. pict nlp @xcite they implemented the xlm's masked language model (mlm) for unsupervised learning. they trained it only using the monolingual data provided by the organizers and the opus project. finally, they also applied xlm preprocessing to the data before training. for supervised learning, they trained language models such as lstm and attention based transformer models with the help of the fairseq library. they trained it using monolingual data provided by the organizers. they applied the inbuilt tokenization provided by fairseq on the data. ## system results tables 2 to 7 list the results of the submitted systems in terms of bleu and chrf2 for all systems, and additionally bert scores for those translating into german. for the bert scores, we list both bert f and bert f with idf weighting to give less weight to commonly occuring words. the ordering of the systems is consistent across all metrics. the supervised systems obtain higher results than the unsupervised systems. the language pair dsb ↔ hsb obtained comparatively high scores for both supervised and unsupervised translation which is very probably due to the high similarity between the two languages. overall, we see no winner across all tasks: huaweitsc has the best scores across all supervised translation tasks, followed by muni-nlp for the de-to/from-dsb and dsb-to/from-hsb translations. these two language pairs only have comparably small parallel data sets which are, notably, the sole basis of muni-nlp's submissions. for the unsupervised translation (where muni-nlp did not participate), aic has the strongest results with the exception of dsb-to/from-de, where huaweitsc is leading. ## conclusion in the wmt 2022 shared task on unsupervised and very low resource mt, we provided the participants with resources for all possible translation directions for the three languages upper sorbian, lower sorbian and german, of which upper sorbian ↔ lower sorbian is a new language pair in comparison to last year's shared task. the participating teams submitted strong systems relying on a wide range of methods. using modeling techniques such as pre-training on parallel data of related languages is important, as is the creation of synthetic data for which we saw the application of different methods. however we also saw that careful modeling on a small data set only can lead to good results. we hope that this shared task will continue to increase the interest in research on methods for under-resourced languages, both for supervised and unsupervised approaches.