Overview of the Shared Task on Machine Translation in D ravidian Languages (2022)

Abstract: This paper presents an outline of the shared task on translation of under-resourced Dravidian languages at DravidianLangTech-2022 workshop to be held jointly with ACL 2022. A description of the datasets used, approach taken for analysis of submissions and the results have been illustrated in this paper. Five sub-tasks organized as a part of the shared task include the following translation pairs: Kannada to Tamil, Kannada to Telugu, Kannada to Sanskrit, Kannada to Malayalam and Kannada to Tulu. Training, development and test datasets were provided to all participants and results were evaluated on the gold standard datasets. A total of 16 research groups participated in the shared task and a total of 12 submission runs were made for evaluation. Bilingual Evaluation Understudy (BLEU) score was used for evaluation of the translations.

## introduction the results of the shared task on machine translation (mt) of dravidian languages held as a part of dravidianlangtech-2022 workshop have been presented in this paper. five translation sub-tasks featured in this shared task, namely: kannada to tamil, kannada to telugu, kannada to sanskrit, kannada to malayalam and kannada to tulu. we evaluated the performance of the systems using bleu scores. training, development, and test data used in this shared task have been released publicly. mt is one of the fundamental problems in the area of natural language processing. we hope that this shared task and associated datasets can further research and development of translation technology for under-resourced dravidian languages. related works have been described in section 2. a brief description about dravidian languages and sanskrit are given in section 3 and section 4 respectively. the task description and the datasets have been discussed in section 5. the description of the systems submitted has been given to section 6. lastly, the results and the conclusion have been discussed in section 7 and section 8 respectively. ## related works in the past few years deep learning (dl) based architectures have increasingly been applied to tackle the problem of mt @xcite @xcite . these architectures require large amounts of data during training and this, in turn, makes them unsuitable for application in development of translation systems for under-resourced languages. @xcite ; @xcite demonstrate good performance on translation of under-resourced languages using multilingual mt systems. another noteworthy approach to tackle this problem is the development of universal translation systems @xcite . the key idea driving this line of research is the development of a system that's capable of transferring linguistic attributes across data from different languages. this is aimed at alleviating the need for large bilingual datasets for under-resourced languages. data augmentation is another approach that has been explored in building of translation systems of under-resourced languages. @xcite propose a framework for a translation system that uses monolingual target side dataset along with pivots grounded in a third high resource language. precisely, they propose a two-stage framework based on pivoting to convert data from high-resourced languages to under-resourced languages, thus augmenting the available data for the translation underresourced languages. another avenue of interest that has been popular amongst researchers working in this domain is application of transfer learning (tl) based approaches to improve the performance of mt systems for under-resourced languages. @xcite train a model for under-resourced mt by initializing some parameters of the model with pa-rameters from a neural model trained on the task of mt for a resource rich language pair. they report an average increase in performance by 5.6 bleu. @xcite demonstrate improved performance on translation of under-resourced languages by employing a simple tl based approach wherein they train a parent model for mt of a resource rich language pair followed by fine-tuning on an under-resourced language pair. it is interesting to note that the authors report improved performance even if the languages in the under-resourced setting are altogether different from the languages which are used to train the model. mahata et al. (2020) study the impact of languages and their relative position in the language family on the performance of tl systems. furthermore, they try to quantify the impact of shared vocabulary on the performance of such systems. in the past few years mt of indian languages has gained increasing traction from the research community. chakravarthi et al. (2019, 2021 ## dravidian languages dravidian languages, which make up the fifth largest linguistic family in the world, are spoken by around 200 million people in south asia and diaspora communities around the world. in dravidian language family, there are 26 languages, including tamil, malayalam, kannada, and telugu, which are considered as major languages, in addition to 20 non-literary languages @xcite . since the most dravidian languages have their writing script, they have a separate block in the unicode computing industry standard @xcite . all of these languages use left-toright writing systems and maintain similar features in their word formation and sentence structure. in these languages, sentences are constructed by a sequence of words and words are formed by adding prefixes and/or suffixes to the root word @xcite @xcite @xcite @xcite @xcite . dravidian languages follow an alpha-syllabic writing scheme, with each character being called a syllable. consonant ligatures are formed when vowels and consonants are tied together with grammar @xcite . tamil was the first language to be listed as a classical language of india and is one of the longestsurviving classical languages of india. being a scheduled language by the indian constitution, it is an official language of tamil nadu, a state of india and puducherry, a territory of india. further, it is also considered as one of the official languages of sri lanka and singapore. besides kerala, karnataka, andhra pradesh, telangana, and the union territory of andaman and nicobar islands, tamil is spoken by significant minorities in four other south indian states. tamil script was first recorded in 580 bce on pottery from keezhadi, sivagangai, and madurai districts of tamil nadu, india by the tamil nadu state department of archaeology and archaeological survey of india @xcite . the script was known as tamili or tamil-brahmi foot_1 . the alphabets of tamil consist of 18 consonants, 12 vowels, and 216 compound letters followed by a special character making total of 247 letters @xcite . tamil is an official language of tamil nadu, sri lanka, singapore, and the union territory of puducherry in india. significant minority speak tamil in the four other south indian states of kerala, karnataka, andhra pradesh, and telangana, as well as the union territory of the andaman and nicobar islands @xcite @xcite @xcite . it is also spoken by the tamil diaspora, which may be found in malaysia, myanmar, south africa, the united kingdom, the united states, canada, australia, and mauritius. tamil is also the native language of sri lankan moors. tamil, one of the 22 scheduled languages in the indian constitution, was the first to be designated as a classical language of india @xcite @xcite . tamil is one of the world's longest-surviving classical languages. the earliest epigraphic documents discovered on rock edicts and "hero stones" date from the 6th century bc. tamil has the oldest ancient non-sanskritic indian literature of any indian language @xcite . malayalam belongs to the dravidian language family and is highly agglutinative. it originated during the last quarter of the 9th century a.d @xcite . as a result of the steep western ghats separating the dialect from the main speech group in the 16th century, it gradually developed into a separate language. the ramacaritam is the first literary work written in malayalam, a combined language of tamil and sanskrit, utilizing the tamil grantha script used in tamil nadu for the writing of sanskrit and foreign words @xcite . there are 13 vowels, 36 consonants, 5 chillu, an anuswara, a visarga, and a chandrakkala making total of 57 letters in malayalam @xcite . telugu belongs to the dravidian language family and is predominantly spoken by the people of andra pradesh. it is the official language of andhra pradesh and telangana with more than 2.75 million telugu speakers 3 . inscriptions of telugu date back to 575 ce. there is a total of 52 letters in telugu with 16 vowels and 36 consonants and the script is called abugida which belongs to the brahmi family 4 . kannada is the second-oldest dravidian language, spoken primarily by residents of karnataka. there are around 44 million kannada speakers worldwide, with over 12.6 million non-kannada speakers in karnataka speaking it as a second or third language foot_2 . it is one of the scheduled languages of the indian constitution, as well as the official and administrative language of karnataka, india. it uses the brahmi script, which comprises 49 letters in total, comprising 13 vowels, 2 diphthongs, and 34 consonants 6 . kannada has a large number of articles, although they are not all digitized. tulu is a prominent dravidian language spoken primarily by the people of dakshina kannada and udupi in karnataka state, as well as some parts of kasaragod in kerala state. tulu is spoken by around 2.5 million individuals who believe it to be their mother tongue 7 . with its particular sociocultural quali ## sanskrit the sanskrit language has been around for hundreds of years, and it uses the devanagari @xcite . with its extensive vocabulary, phonology, grammar, and syntax, sanskrit literature has a long history of use in ancient poetry, drama, science, and philosophy @xcite . it consists of 16 vowels and 36 consonants and belongs to the indo-european language family. sanskrit is a highly inflected language divided into eight chapters to make it more structured and understandable (panini asthadhyayi) @xcite . despite the enormous number of articles, the quantity of digital resources is limited, especially for the parallel corpus. objective of the task was to train/develop mt systems for the language pairs that were provided. participants translated the test data using mt models proposed by them and submitted the results to the workshop organizers. bleu is selected as the evaluation metric to evaluate the submitted mt models. in order to determine the participants' rank, the submissions were compared with gold-standard data. ## system description out of 16 research groups, 12 run submissions were made by 4 teams. set ii received the maximum number of submissions (4 teams) followed by set i (3 teams). further, results of the participated systems in terms of bleu score and system ranks for each language pair are shown in ## results and discussion as shown in ## conclusion the shared task on mt in dravidian languages opened up a slew of new research opportunities in the field of mt in dravidian languages. the task also involves sanskrit, an ancient language, in addition to dravidian languages. despite positive reactions and enthusiasm for attending the event, the number of system submissions was not impressive. we collected kannada-tamil, kannada-malayalam, and kannada-telugu from samanatar, a collection of parallel corpora. further, kannada-sanskrit and kannada-tulu parallel corpora were created manually. the performance and bleu scores of the participants are not credible, yet they are not discouraging. the main inference from the participants' results is that along with the baseline mt models, efficient dataset preparation methods, namely, backtranslation and subword tokenization also necessary to achieve better performance in the translation of morphologically rich languages. as a final note, we hope to continue conducting this workshop in the coming years to contribute to the advancement of language technology for underresourced dravidian languages.