Findings of the L o R es MT 2020 Shared Task on Zero-Shot for Low-Resource languages (2020)

Abstract: This paper presents the findings of the LoResMT 2020 Shared Task on zero-shot translation for low resource languages. This task was organised as part of the 3rd Workshop on Technologies for MT of Low Resource Languages (LoResMT) at AACL-IJCNLP 2020. The focus was on the zero-shot approach as a notable development in Neural Machine Translation to build MT systems for language pairs where parallel corpora are small or even non-existent. The shared task experience suggests that back-translation and domain adaptation methods result in better accuracy for small-size datasets. We further noted that, although translation between similar languages is no cakewalk, linguistically distinct languages require more data to give better results.

## introduction research and development in statistical and neural machine translation has rapidly emerged over in the last one decade especially after the availability of several open source machine translation (mt) toolkits like: moses @xcite , opennmt @xcite , nematus @xcite , marian @xcite , etc. for the past few years, researchers, developers, users and commercial organizations are widely using neural machine translation (nmt) @xcite @xcite @xcite to enhance the performance of their mt systems. nmt has become the preferred paradigm due to its ability to produce better results. despite the multiple advantages of using nmt and statistical machine translation (smt) methods, several challenges are also encountered, the main challenge being the lack of quality data for training the systems. both smt and nmt require large-sized parallel corpora. however, out of 7,117 languages, foot_0 most languages are low resourced or even endangered. this particular challenge has long posed a strong impediment for developing nmt systems for low resource languages @xcite . overcoming this obstacle is an extensive and tedious process. the preparation of a parallel corpus in any language can be a very costly procedure that demands the expertise of language professionals at several levels. it is therefore necessary to exploit the existing resources to build mt systems in low resource languages. methodologies such as data augmentation, exploitation of monolingual data, cross-lingual transfer etc. are preferred approaches under the aforesaid circumstances. in the preceding year, a shared task @xcite was organised where a monolingual and parallel corpus for the low-resource languages bhojpuri, magahi, sindhi, and latvian was provided to create nmt/smt systems foot_1 . this year's workshop in an extension to the same objective as last year, but this time the team has focused on the zero-shot approach @xcite for building quality mt systems. in addition to pivot mt, the zero-shot approach is one notable development in nmt to build mt systems for language pairs where parallel corpora are small or even nonexistent. however, the performance of zero-shot nmt is low compared to pivot mt in general. in this paper, we discuss the results of the loresmt 2020 shared task, organised as part of the 3 rd workshop on technologies for mt of low resource languages (loresmt) foot_2 at aacl-ijcnlp 2020 4 . in this task, we solicited participants to submit novel zero-shot nmt systems for the following language pairs: the remaining paper is organized as follows. section 2 presents the setup and schedule of the loresmt 2020 shared task and section 3 presents the dataset used in the competition. section 4 describes the approaches used by participants of the competition and section 5 presents and analyzes the results they obtained. finally, 6 concludes this paper and presents avenues for future work. ## task setup and schedule based on a detailed call for participation, researchers were asked to register themselves. the choice of language pair was left to the participants. these registered participants were sent the links to the training (train) dataset including monolingual and development (dev) data, along with a description of the format and statistics of the dataset. they were allowed to use only additional monolingual data to train the system, with the condition that the additional monolingual dataset should be publicly available. moreover, participants were allowed to use pretrained word embeddings, and publiclyexisting linguistic models. the participants were given 24 days to experiment and develop a system. after this period, the test set was released and the participants had 5 days to test and upload their system using the following abbreviations: the complete timeline of the shared task is given in ## datasets the dataset of this shared task comprises three domains: news, subtitling and/or literature. details of the collected sources are described below: the participants of the shared task were provided with more than one million words of monolingual data for each language pair, while 500 manually translated and validated parallel sentences were provided for dev and test set. the complete shared task datasets are available at github foot_5 . the detailed statistics of the dataset in each language is provided in ## participants and methodology a total of 5 participants registered for the shared task, with most of the teams registering to participate for hindi↔bhojpuri and hindi↔magahi language pair except 1 team (see next, we give a short description of the approach taken by each team for building their system(s). more details about the approaches can be found in the papers submitted by the respective teams. ## results as previously mentioned, the participants were allowed to use monolingual datasets, other than that provided. however, due to the lack of a similar substitute monolingual dataset for bhojpuri and magahi, participants used only one of the provided data by the shared task organisers. the nlprl team used orthographically similar hindi-nepali data to build their system. on the other hand, the cnlp-nist team only used additional hindi and russian monolingual data for the constrained system submission. as mentioned earlier, for the evaluation of the system, 500 sentences were given to the participants in each language pair for each direction. the results of the participating teams on hindi-bhojpuri, hindi-magahi and hindi-russian language pairs is presented in ## conclusion in this paper, we have reported the findings of the loresmt 2020 shared task on zero-shot translation for low resource languages, organized as part of the 3 rd loresmt workshop at aacl-ijcnlp 2020. all the systems submitted used the unsupervised method. we conclude that the use of domain adaptation and back translation methods provides better results for mt system training where the datasets are small-sized. another concluding point is that the masked sequence-to-sequence pre-training method provides comparatively low performance on all measures: bleu, precision, recall, f-measure and ribes. bhojpuri to hindi has provided better accuracy scores than vice versa for both the teams who selected bhojpuri and hindi as their language pairs. the systems trained for hindi and russian did not provide desired results in any language direction, despite them having larger datasets than the other two languages in the shared task. this understanding should be accompanied with the knowledge that russian and hindi are completely dissimilar languages belonging to separate language families. we also believe that a human evaluation could provide better insights than automatic evaluation metrics. in the next version of the shared task, we are planning to introduce human evaluation of the systems, in order to extend and improve the findings of our shared task on low resource languages.