Findings of the WMT 2020 Shared Tasks in Unsupervised MT and Very Low Resource Supervised MT (2020)

Abstract: We describe the WMT 2020 Shared Tasks in Unsupervised MT and Very Low Resource Supervised MT. In both tasks, the community studied German to Upper Sorbian and Upper Sorbian to German MT, which is a very realistic machine translation scenario (unlike the simulated scenarios used in particular in much of the unsupervised MT work in the past). We were able to obtain most of the digital data available for Upper Sorbian, a minority language of Germany, which was the original motivation for the Unsupervised MT shared task. As we were defining the task, we also obtained a small amount of parallel data (about 60000 parallel sentences), allowing us to offer a Very Low Resource Supervised MT task as well. Six primary systems participated in the unsupervised shared task, two of these systems used additional data beyond the data released by the organizers. Ten primary systems participated in the very low resource supervised task. The paper discusses the background, presents the tasks and results, and discusses best practices for the future.

## introduction there is no machine translation available for most of the approximately 7000 languages spoken on the planet earth. this is because very limited or no parallel corpora are available. research on unsupervised and very low resource machine translation is important for alleviating this problem. unsupervised machine translation requires only monolingual data, while very low resource supervised machine translation uses very limited parallel data. at wmt 2018 and wmt 2019, the first shared task @xcite and second shared task @xcite on unsupervised machine translation (umt), were held as part of the news translation track. in both 2018 and 2019 the scenarios simulated low resource setups using medium or high resource languages. in 2018, the language pairs were turkish-english, estonian-english and german-english. in 2019, we tested unsupervised systems for german to czech unsupervised translation (where no german/czech parallel data was allowed). in the 2020 shared task we proposed a third edition on umt, which aimed at a more realistic scenario, german to upper sorbian (and upper sorbian to german) translation. upper sorbian is a minority language of germany that is in the slavic language family (e.g., related to lower sorbian, czech and polish), and we provide here most of the digital data that is available for upper sorbian, as far as we know. the amount of monolingual data available for upper sorbian is quite small (see below), making unsupervised machine translation very challenging. while working on the umt task we were able to obtain a very small amount of parallel data (about 60000 parallel sentences) for this language pair, which allowed us to additionally offer the very low resource supervised translation task. the exact tasks studied in the shared tasks were: the data used can be downloaded from the shared task webpage. 1 this paper will give some background for the task, discuss the data made available (with a particular focus on considerations for benchmarks for future umt and very low resource mt research), discuss how the tasks went with a presentation of the results, and then conclude. ## background the 2020 shared tasks focused on the very low resource language upper sorbian (a slavic minority language spoken in the eastern part of germany). the sorbian language (referring to upper sorbian and lower sorbian) has a special protected status in the "grundgesetz" of germany (similar to a constitution). working with the sorbian institute foot_1 we initially prepared an unsupervised mt task. we expect this task to become a standard benchmark for unsupervised mt development. this task particularly relies on having high quality upper sorbian monolingual data, which we obtained from the sorbian institute and the witaj sprachzentrum. we also offered data which lmu munich obtained through web crawling. finally, the sorbian institute has also provided medium quality data which has not been quality checked. the witaj sprachzentrum (witaj language center foot_2 ) then provided a small training corpus of german/upper sorbian parallel data, which was used in the very low resource supervised machine translation task. we expect this task to become a standard task for very low resource scenarios, and note that the results are important as they will directly inform efforts to create state-of-the-art machine translation systems for use by the upper sorbian community. the witaj sprachzentrum provided development and "development test" (not blind test) sets for german to upper sorbian and upper sorbian to german translation. the development set is used to tune parameters, while the devtest set is used to measure progress using automatic metrics. the witaj sprachzentrum also provided the blind test sets, which were released just in time for the evaluation. we note that cis (lmu munich), the sorbian institute and the witaj sprachzentrum hope to organize a task for unsupervised lower sorbian translation in the near future. lmu munich would be interested in adding new typologically diverse languages as unsupervised and/or very low resource tasks in future wmt shared tasks. ## basic tasks and evaluation we provide some basic details about the tasks and evaluation here. unsupervised mt: unsupervised translation from german to upper sorbian, and unsupervised translation from upper sorbian to german. we allow the use of all german data released for wmt, except that the german side of the small parallel german/upper sorbian training corpus may not be used. all upper sorbian data we release may be used. no other language data may be used. very low resource supervised mt: supervised translation from german to upper sorbian, and supervised translation from upper sorbian to german. we allow the use of all german and upper sorbian data released for wmt, including the small parallel german/upper sorbian training corpus. other wmt data for other languages may also be used. upper sorbian is a slavic language which has strong similarities to czech, so the german/czech data we discuss below may be of particular interest for multilingual systems. we used automatic metrics for the evaluation of this task. we believe that manual evaluation may not be so necessary for unsupervised mt and very low resource mt development, because automatic metrics worked well at relatively low translation quality levels in the past. translation to upper sorbian would have been fairly difficult to evaluate in a human evaluation, as we do not have easy access to a large number of native speakers. ## data we describe the data released for the two scenarios, unsupervised and very low resource supervised. the unsupervised mt scenario allowed the use of all german data released for wmt, except that the german side of the small parallel german/upper sorbian training corpus could not be used. all upper sorbian data available on the web page was allowed. no other language data could be used (no parallel, no other monolingual data sets for any language except those explicitly listed as usable for unsupervised). the very low resource supervised mt scenario allowed the use of all german and upper sorbian data released for wmt, including the 60000 sentence parallel german/upper sorbian training corpus. other wmt 2020 data for other languages were also allowed. upper sorbian is a slavic language which is related to czech, so the ger-man/czech parallel data we briefly describe next were of particular interest for building multilingual systems. we would also like to express our thanks to the opus project for the german/czech parallel data. in addition to training data, we also provided development data. specifically, we provided development and test sets for german to upper sorbian and upper sorbian to german. these were usable for both unsupervised and very low resource. the dev set was intended for use for parameter tuning (and the participants were asked to not use it as a parallel training corpus). the test set was intended for system evaluation during development (likewise, the participants were asked to not use it as a parallel training corpus). we would like to thank jindřich libovický for creating the data splits and for work on the training data. the specific data sets are presented in the translation output was submitted as real case, detokenized, and in sgml format. we used the matrix for submission, thanks to barry haddow for assistance. the standard script wrap-xml.perl was used to create the sgm files for upload. ## results in this section we discuss the results for the unsupervised track, a non-constraint system that does not use parallel german/sorbian data but does use german-english parallel data (which does not meet the constraints for the unsupervised task), and the very low resource task. we note that the absolute bleu scores are surprisingly high, this may be due to unusually homogeneous train and test sets. ## analysis overall we proposed two new benchmark tasks for low resource mt. the unsupervised mt track can be used as a benchmark for testing new unsupervised mt approaches. the very low resource track can be used as a benchmark for testing mt systems when one language has very little parallel data. additionally, one system which was conceptually unsupervised (in that it used no parallel german-upper sorbian data), effectively defined a new benchmark, which we will call unsupervised mt with multilingual transfer, by using additional english-german parallel data, see below. dev and test sets devtest.tar.gz the participants were requested to please use dev to tune system parameters, and test to measure progress. these files were allowed in both tracks. german/czech parallel data this data was not allowed for unsupervised. for very low resource mt we allowed all german/czech parallel corpora obtainable from the opus project. the de-cs corpora we particularly recommended using are: europarl v8 and jw300 v1. these two corpora may be somewhat similar to the de-hsb parallel training and test data, but this is far from certain. the unsupervised mt paradigm. in practice one could create a "very very low resource" system by simply training a supervised system on this data. we however argue that system developers need to have access to standard dev and devtest sets in order to measure progress. having access to a dev set simulates having access to, e.g., a native speaker of the low resource language, who is able to look at outputs during mt development and judge the changes in quality obtained through simple hyperparameter changes. in our view the use of a devtest set to measure bleu (which is completely standard in all unsupervised mt research) is similar to this, and of course also represents a small amount of available parallel data. a further argument for our view is that having a designated dev set for parameter tuning strongly helps with replicability of results. however, it is important in our view that this data not be trained on (in terms of using it as a parallel corpus to training a supervised system). the restriction of the rest of the data to be monolingual and carefully controlled makes sense for the straightforward testing of bilingual unsupervised systems, but we describe an interesting different multilingual scenario next. unsupervised mt with multilingual transfer: sjtu-nict submitted a system to unsupervised mt which looked at the reasonable scenario of assuming that the high resource language (german for german-upper sorbian) has parallel corpora with another language. in their system they used an english-german parallel corpus to initialize their german-upper sorbian unsupervised mt system. this makes their system conceptually a multilingual system. another obvious choice would be to try german-czech and/or ger-man paired with another slavic language (and in fact, in the very low resource track, this was a common strategy). there were no competing systems for this unplanned benchmark (which was essentially created by sjtu-nict's submission), but we suggest that this result is also interesting for comparison in future research and intend to offer a new track for this scenario in the shared task next year. very low resource mt: please see the shared task system descriptions to see the wide variation in the exact data used for the very low resource task. there was obviously a high level of interest overall in this task, and we already have plans to offer more tasks of this kind as well. this task is very interesting as even if an unsupervised mt system is actually deployed for a particular language pair, the users of the system can post-edit the output of that system and will soon therefore be interested in training higher quality supervised systems using the very low resource scenario. the parallel data used in this track was in fact created manually by the witaj sprachzentrum in order to be used by the upper sorbian community in germany (to train the baseline moses system we mentioned in the results section, and for future neural machine translation work which will be informed by the results produced by the research community). as the result of the existence of this pipeline more data may become available in the future. ## conclusion the wmt 2020 shared tasks in unsupervised mt and very low resource supervised mt have created three interesting new benchmarks for fu-ture research. in addition to the initially proposed benchmark on straight-forward bilingual unsupervised mt, we suggest the use of the unsupervised multilingual transfer result as an additional new benchmark. the very low resource mt benchmark also generated strong participation from the research community. we hope that additionally we have played a role in raising the interest of the nlp community in upper sorbian language processing, and that this interest will extend to lower sorbian language processing and in fact extend to working on other understudied languages as well. we plan to organize similar tasks for the three benchmarks next year. options include, e.g., more parallel data for upper sorbian, an unsupervised task for lower sorbian, and more ambitiously the inclusion of new typologically diverse languages. please don't hesitate to contact us if you are interested, particularly if you have access to appropriate data.