Findings of the Shared Task on Machine Translation in D ravidian languages (2021)

Abstract: This paper presents an overview of the shared task on machine translation of Dravidian languages. We presented the shared task results at the EACL 2021 workshop on Speech and Language Technologies for Dravidian Languages. This paper describes the datasets used, the methodology used for the evaluation of participants, and the experiments’ overall results. As a part of this shared task, we organized four sub-tasks corresponding to machine translation of the following language pairs: English to Tamil, English to Malayalam, English to Telugu and Tamil to Telugu which are available at https://competitions.codalab.org/competitions/27650 . We provided the participants with training and development datasets to perform experiments, and the results were evaluated on unseen test data. In total, 46 research groups participated in the shared task and 7 experimental runs were submitted for evaluation. We used BLEU scores for assessment of the translations.

## introduction in this paper, we present the results of the shared task on machine translation of dravidian languages of the workshop on speech and language technologies for dravidian languages held at eacl 2021. the shared task features four sub-tasks: a translation task from english to tamil, english to telugu, english to malayalam and tamil to telugu. they all closely related languages and are underresourced now. they are mutually intelligible since speakers of dravidian (tamil) languages can readily understand each other without prior familiarity or special effort @xcite . the performance of our tasks was evaluated using automatic measures bleu @xcite . this shared task's primary objectives are to further state of the art in machine translation of low resource languages belonging to the dravidian (tamil) language family. the training data, development data and test data, and results are available publicly foot_0 . we hope the datasets released as a part of this task will contribute positively towards forwarding research in the machine translation of under-resourced languages. ## related work machine translation of under resource languages is an open and an active research area. in this day and age when translation systems are increasingly being built upon neural network-based architectures @xcite @xcite , the development of such systems for under-resourced languages is a challenging task due to the lack of availability of resources. multilingual extensions to these architectures have been proposed @xcite @xcite which have been shown to improve on low-resource languages. recent studies have also extended this to a massively multilingual setting @xcite . @xcite demonstrated a transfer learning-based approach by utilizing shared lexical and sentence level representations across multiple source languages, thereby developing a system that performs well in low resource scenarios. @xcite propose a general framework for data augmentation in low-resource machine translation that not only uses target-side monolingual data but also pivots through a related high-resource language hrl. @xcite 's key idea is first to train a high-resource language pair (the parent model), then transfer some of the learned parameters to the low-resource pair (the child model) to initialize and constrain training. @xcite propose a transfer learning-based method, wherein they train a "parent" model with a high-resource language pair followed by which they train on a "child" model on a corpus of a low-resource language pair. it was observed that this model is better than the models trained on just the low-resource languages. @xcite propose a transfer learning-based method, wherein they train a parent model trained on a high-resource language pair, followed by which they transfer some of parameters to a child model that they subsequently train on the low-resource language pair. lakew et al. (2017) leverage the use of duality of translations generated by the system for zero-shot; these translations are used along with the parallel data to train the neural network. @xcite show the results of the loresmt 2020 shared task. this workshop was held along with aacl and reported good bleu scores in case of the low-resource bhojpuri-hindi language pair. @xcite focussed on the translation of the low resource language pairs nepali-english and sinhala-english. they reported the results of these translation tasks for statistical as well as phrase-based methods. chakravarthi et al. (2019c) created a multimodal corpora for dravidian languages. m (2013) developed a statistical machine translation system for english-tamil using transfer based on computational linguistics. @xcite proposed a factored statistical machine translation system for english to the tamil language. @xcite conducted the machine translation for indian languages shared task in 2017. as a part of this shared task, organizers have released the parallel corpora for english-tamil and english-malayalam. for dravidian language translation, chakravarthi et al. @xcite ) created a machine translation to improve wordnet entries. @xcite demonstrate a transfer learning based translation system and a divergence index to calculate the success rate of the system. @xcite propose a multi-layer neural network based approach wherein they use n-grams extracting from connecting phrases of the phrase table for the task of machine translation on kannada-telugu language pair. chakravarthi et al. @xcite ; chakravarthi (2020) studied the utilisation of orthographic information to improve the machine translation for dravidian languages. ## dravidian languages tamil is an official language in tamil nadu, puducherry, sri lanka and singapore @xcite . tamil was the first to be listed as a classical language of india, one of 22 scheduled languages in india's constitution, and is one of the world's longest-surviving classical languages. in jain samavayanga sutta (3rd-4th century bce) and pannavana sutta, a script called damili (tamili) is listed as the seventeenth of eighteen scripts in india, an early mention of a script for writing the tamil language @xcite . similarly, lipisala samdarshana parivarta, the tenth chapter of the lalitavistara sutta (3rd century ce) a sanskrit text, mentions siddhartha (later gautam buddha) learning dravida-script (tamil script) and dakshinya-script (southern tamil script) along with other sixty two scripts. tamil was called damil in jain prakrit, dramil in buddhist pali, and dravida in sanskrit @xcite . the tamil scripts are first attested in the 580 bce as tamili foot_1 script inscribed foot_2 on the pottery of keezhadi, sivagangai and madurai district of tamil nadu, india (sivanantham and seran, 2019) foot_3 by tamil nadu state department of archaeology and archaeological survey of india. the writing systems of tamili was explained in the old grammar text tolkappiyam dates between 8000 bce to 580bce @xcite @xcite @xcite . malayalam was tamil's west coast dialect until about the 15th-century ce @xcite . the dialect gradually developed into a different language in the 16th century @xcite due to geographical separation by the steep western ghats from the main speech group. ramacaritam ('deeds of rama') was the first literary work in malayalam written in malayalam, a combination tamil and sanskrit, using the tamil grantha script which was used in tamil nadu to write sanskrit and foreign words @xcite . telugu existed in the earliest time in the form of inscriptions from 575 ce onwards. telugu literary works split from tamil by the first grammar of telugu in the 13th century ce which was written by atharvana acharya, naming it the trilinga śabdānusāsana (or trilinga grammar). similarly, kannada also split from tamil by 8th century ce. the kappe arab-team languages bleu gx @xcite english-telugu 38.86 irlab-daiict @xcite ) english-telugu 6.25 mucs shared task @xcite english-telugu 0.29 gx @xcite english-tamil 36.66 spartans english-tamil 28.27 irlab-daiict @xcite english-tamil 6.04 mucs shared task @xcite english-tamil 1.66 gx @xcite english-malayalam 19.84 spartans english-malayalam 15.31 irlab-daiict @xcite english-malayalam 8.43 mucs shared task @xcite english-malayalam 0.48 gx @xcite tamil-telugu 35.29 mucs shared task @xcite tamil-telugu 0.43 irlab-daiict @xcite tamil-telugu 0.0 the shared task was hosted on codalab. four translation sub-tasks were organized as a part of this task: english to tamil, english to malayalam, english to telugu and tamil to telugu. participants were given a choice to participate in the sub-tasks they wanted to. training, development and test datasets of parallel sentences for each language pair were provided to all the participants. the task was to train/develop machine translation systems for the given languages. for evaluation, the participants translated the test data using the translation systems and the results were submitted to the organizers of the workshop. the submissions were evaluated by comparing them with the gold standard test set translations available, for which bleu scores were used as the metric to rank the participants and subsequently the results were returned to the participants. ## system description a summary of the results of the shared task can be found in tables 1. to evaluate the performance of the submitted systems, we calculated bleu scores for each of these systems. we have listed out the short description of participants systems, for more details please refer their paper. ## results and discussion based on the results reported for the test sets by the top 3 teams in each language translation sub-task as shown in using the system descriptions along with information from tables 1 and 2 , the system rankings can be summarized as follows: the reason for the variation in results among language pair tasks is due the following reasons as reported in the system descriptions: ## conclusion this paper described the shared task of machine translation of dravidian (tamil) languages to be presented at the first workshop on speech and language technologies for dravidian technologies and summarized the results of this workshop. the best performing systems submitted to this workshop achieved good performance in terms of bleu scores inspite of the lack of data available for training. this is a promising result in that such similar techniques can be applied to other under resourced languages. we would like to conclude by saying that we hope to continue to conduct this workshop over the coming years, and therefore continue to contribute to the development of language technology for under-resourced dravidian (tamil) languages.