Findings of the L o R es MT 2021 Shared Task on COVID and Sign Language for Low-resource Languages (2021)

Abstract: We present the findings of the LoResMT 2021 shared task which focuses on machine translation (MT) of COVID-19 data for both low-resource spoken and sign languages. The organization of this task was conducted as part of the fourth workshop on technologies for machine translation of low resource languages (LoResMT). Parallel corpora is presented and publicly available which includes the following directions: English↔Irish, English↔Marathi, and Taiwanese Sign language↔Traditional Chinese. Training data consists of 8112, 20933 and 128608 segments, respectively. There are additional monolingual data sets for Marathi and English that consist of 21901 segments. The results presented here are based on entries from a total of eight teams. Three teams submitted systems for English↔Irish while five teams submitted systems for English↔Marathi. Unfortunately, there were no systems submissions for the Taiwanese Sign language↔Traditional Chinese task. Maximum system performance was computed using BLEU and follow as 36.0 for English–Irish, 34.6 for Irish–English, 24.2 for English–Marathi, and 31.3 for Marathi–English.

## introduction the workshop on technologies for machine translation of low resource languages (loresmt) foot_0 is a yearly workshop which focuses on scientific research topics and technological resources for machine translation (mt) using low-resource languages. based on the success of its three predecessors @xcite @xcite , the fourth loresmt workshop intoduces a shared task section based on covid-19 and sign language data as part of its research objectives. the hope is to provide assistance with translation for low-resource languages where it could be needed most during the covid-19 pandemic. to provide a trajectory of the loresmt shared task success, a summary of the previous tasks follows. the first loresmt shared task @xcite took place in 2019. there, monolingual and parallel corpora for bhojpuri, magahi, sindhi, and latvian were provided as training data for two types of machine translation systems: neural and statistical. as an extension to the first shared task, a second shared task @xcite was presented in 2020 which focused on zero-shot approaches for mt systems. this year, the shared task introduces a new objective focused on mt systems for covidrelated texts and sign language. participants for this shared task were asked to submit novel mt systems for the following language pairs: • english↔irish • english↔marathi the rest of our work is organized as follows. section 2 presents the setup and schedule of the shared task. section 3 presents the data set used for the competition. section 4 describes the approaches used by participants in the competition and section 5 presents and analyzes the results obtained by the competitors. lastly, in section 6 a conclusion is presented along with potential future work. ## shared task setup and schedule this section describes how the shared task was organized along with the systems. registered participants were sent links to the training, development, and/or monolingual data (refer to section 3 for more details). they were allowed to use additional data to train their system with the condition that any additional data used should be made publicly available. participants were moreover allowed to use pre-trained word embeddings and linguistic models that are publicly available. as a manner of detecting which data sets were used during training, participants were given the following markers for denotation: each team was allowed to submit any number of systems for evaluation and their best 3 systems were included in the final ranking presented in this report. each submitted system was evaluated on standard automatic mt evaluation metrics; bleu @xcite , chrf @xcite and ter @xcite . the schedule for deliver of training data and release of test data along with notification and submission can be found in in this section, we present background information about the languages and data sets featured in the shared task along with a itemized view of the linguistic families and number of segments in ## participants and methodology a total of 12 teams registered for the shared task: 5 teams registered to participate for all language pairs, 5 teams registered to participate only for english↔marathi, one team registered for taiwanese↔mandarin (traditional chinese) sign language and one team registered for english↔irish. out of these, a total of 6 teams submitted their systems on covid while none of them submitted a system for sign language. out of the submitted systems, two teams participated for the english↔irish and english↔marathi tasks, one team participated for english-irish and three teams participated for english↔marathi (see team english-irish english-marathi tsign-tchinese system description paper iiitt en2ga & ga2en en2mr & mr2en -(puranik et al., 2021) onenlp-iiith -en2mr & mr2en -(mujadia and sharma, 2021) a3108 -en2mr & mr2en -(yadav and shrivastava, 2021) cfilt-iitbombay -en2mr & mr2en -(jain et al., 2021) ucf en2ga & ga2en en2mr & mr2en -(chen and fazio, 2021) adapt dcu en2ga --(lankford et al., 2021) total 3 5 0 6 next, we give a short description of the approaches used by each team to build their systems. more details about the approaches can be found in the papers by respective teams in the accompanying proceeding. the model can perform 11 languages: assamese, bengali, gujarati, hindi, kannada, malayalam, marathi, oriya, punjabi, tamil, telugu pre-trained on the samanantar data set, the largest data set for indic languages during the time of submission. the model is fine-tuned on the training data set provided by the organizers and a parallel bible corpus for marathi. the team used the parallel bible parallel corpus from a previous task (mul-tiindicmt task in wat 2020). after conducting various experiments, the best checkpoint was recorded and predicted upon. for irish, the team fine-tuned an opus mt model from helsinki nlp on the training data set, and then predicted results after recording. after careful experimentation, the team observed that the opus mt model outperformed the other models giving it the highest scoring model award. ## conclusion we have reported the findings of the loresmt 2021 shared task on covid and sign language translation for low-resource languages as part of the fourth loresmt workshop. all submissions used neural machine translation except for the one from onenlp-iiith. we conclude that in our shared tasks the use of transfer learning, domain adaptation, and back translation achieve optimal results when the data sets are domain specific as well as small-sized. our findings show that uni-gram segmentation transfer learning methods provide comparatively low results for the following metrics: bleu, chrf and ter. the highest bleu scores achieved are 36.0 for english-to-irish, 34.6 for irish-to-english, 24.2 for english-to-marathi, and 31.3 for marathi-to-english. in future iterations of the loresmt shared tasks, extended corpora of the three language pairs will be provided for training and evaluation. human evaluation on system results will also be conducted. for sign language mt, the tasks will be fine-grained and evaluated separately.