Findings of the WMT 2023 Shared Task on Low-Resource I ndic Language Translation (2023)

Abstract: This paper presents the results of the low-resource Indic language translation task organized alongside the Eighth Conference on Machine Translation (WMT) 2023. In this task, participants were asked to build machine translation systems for any of four language pairs, namely, English-Assamese, English-Mizo, English-Khasi, and English-Manipuri. For this task, the IndicNE-Corp1.0 dataset is released, which consists of parallel and monolingual corpora for northeastern Indic languages such as Assamese, Mizo, Khasi, and Manipuri. The evaluation will be carried out using automatic evaluation metrics (BLEU, TER, RIBES, COMET, ChrF) and human evaluation.

## introduction low-resource indic languages refer to the vast array of languages spoken in india, which, unfortunately, possess limited linguistic resources available for their study and development. these languages typically suffer from a combination of factors that set them apart from the more prominent and widely supported languages spoken in the country. the challenges these languages face include having a smaller number of speakers, a relative lack of governmental support, inadequate documentation, and limited access to technological resources. india is renowned for its linguistic diversity, with a rich tapestry of languages spoken across the subcontinent. the eighth schedule of the indian constitution officially recognizes 22 languages, which receive significant government backing and protection. however, beyond these major languages, numerous smaller languages and dialects are spoken by various indigenous and minority communities throughout the country. these low-resource indic languages face a series of interconnected challenges that make their preservation and promotion difficult: lack of written scripts, limited vocabulary resources, inadequate linguistic research, and insufficient digital content. the collective impact of these factors makes it challenging to preserve and promote lowresource indic languages. as a consequence, they are at risk of falling into disuse, with their speakers shifting to more widely recognized languages. efforts to document, revitalize, and support these languages are crucial not only for linguistic diversity but also for the preservation of cultural heritage and the rights of minority language communities in india. efforts are being made by various organizations, researchers, and language enthusiasts to address the issues faced by low-resource indic languages @xcite . these initiatives involve language documentation, the development of writing systems, the creation of linguistic resources such as parallel corpora @xcite , parallel fragment extraction from comparable corpora @xcite , dictionaries and grammars, and the promotion of language use through educational programs and digital platforms. technology indeed plays a pivotal role in supporting low-resource indic languages. in recent years, machine learning and natural language processing techniques have been harnessed to create innovative solutions for these languages, including speech recognition, machine translation, and text-to-speech systems. these technological advancements offer a transformative potential in addressing the linguistic challenges faced by these languages and can have a profound impact on their preservation and revitalization. to work towards the goal of supporting lowresource indic languages, we organized the "indic mt shared task" focusing on several lesspopular languages that belong to different language families. these languages include as-samese (indo-aryan), mizo (sino-tibetan), khasi (austroasiatic), and manipuri (sino-tibetan). in this shared task, we present indicne-corp1.0 in which parallel (english-assamese (en-as), english-mizo (en-lus), english-khasi (en-kha), english-manipuri (en-mni)) and monolingual (assamese, mizo, khasi, manipuri) corpora for northeastern indic languages available. ## shared task: low-resource indic language translation in recent years, there has been significant improvement in the performance of machine translation (mt) systems. this progress can be attributed to the development of new techniques, such as multilingual translation and transfer learning. as a result, the benefits of mt are no longer restricted to users of widely spoken languages. this advancement has led to a growing interest within the research community in expanding mt coverage to encompass a wider range of languages, each with its unique geographical presence, degree of diffusion, and level of digitalization. however, despite the enthusiasm for extending mt to more languages and users, there remains a substantial challenge. the challenge stems from the fact that mt methods typically require large volumes of parallel data for training high-quality translation systems. this requirement has proven to be a major hurdle, particularly when dealing with low-resource languages where obtaining such extensive parallel data can be exceedingly difficult. consequently, there is a pressing need to develop mt systems that can perform well even when trained on relatively small parallel datasets. the ability to achieve effective machine translation with limited resources is of paramount importance for increasing accessibility and usability across a wide spectrum of languages and linguistic communities. in this translation task, our focus was on the following language pairs (both directions for each): in this translation task, participants had the opportunity to submit up to 1 primary system for each language pair/translation direction, where no additional parallel data was permitted for training. additionally, participants could submit up to 2 contrastive systems for each language pair/translation direction. this structure allowed participants to showcase their translation systems under various conditions and constraints, including the absence of additional parallel data in the case of primary systems. ## dataset: indicne-corp1.0 in the creation of indicne-corp1.0, we compiled datasets from our prior research projects, including contributions from laskar et al. (2020, 2022); khenglawt et al. (2022); laskar et al. (2021) ; @xcite . these datasets served as the foundation for constructing both parallel and monolingual corpora. in our earlier works, we undertook the development of english-assamese (eng-asm) @xcite , english-mizo (eng-lus) @xcite , english-khasi (eng-kha) @xcite , english-manipuri (eng-mni) @xcite parallel and monolingual corpora for assamese, mizo, khasi and manipuri languages. the different online sources were explored that include bible, multilingual online dictionary (xobdo and glosbe), multilingual question paper, pmindia foot_0 (haddow and kirefu, 2020), web pages, blogs and online news papers. the collected data statistics for parallel (train, validation and test set) and monolingual corpora are presented in subsequent sections below. for primary investigation in this shared task, we have not included very complex sentences in the test set. ## participants and system descriptions in this shared task, a total of 31 teams registered and contributed, as indicated in however, we have received system description papers from 9 teams and included concise system details for those teams where the authors provided such information. cfilt-iitb (gaikwad et al., 2023): the participant utilized phrase-pair injection @xcite , back-translation @xcite , and transfer learning with the help of large pre-trained multilingual indictrans2 model @xcite to build nmt systems for the english-assamese and english-manipuri language pairs. iol research @xcite : the contributor used monolingual data to train two denoising language models similar to t5 @xcite and bart @xcite , and then used parallel data to fine-tune the pre-trained language models to obtain two multilingual machine translation models. besides, the multilingual machine translation models were used to translate english monolingual data into other multilingual data, forming multilingual parallel data as augmented data @xcite to build nmt systems for english-assamese, english-mizo, english-khasi, and english-manipuri language pairs. iacs-lrilt @xcite : the team iacs-lrilt used indicbart @xcite pre-trained language model for fine-tuning the training data to build nmt systems for english-assamese, and english-manipuri language pairs. guit-nlp @xcite : team guit-nlp used back-translation @xcite strategy and explored nmt systems by leveraging subword tokenization @xcite and hyperparameters tuning for english-assamese, english-mizo, and english-khasi language pairs. nits-cnlp (singh et al., 2023): the nits-cnlp team used the opennmt toolkit @xcite and built a transformer-based @xcite nmt model with hyperparameters tuning for the english-manipuri language pair. nict-ai4b @xcite : the group explored nmt systems by leveraging backtranslation strategy @xcite with denoising techniques @xcite and fine-tuned indictrans2 model @xcite for the english-assamese, english-mizo, english-khasi, and english-manipuri language pairs. muni-nlp @xcite : the participant explored transformer-based @xcite nmt systems by investigating different hyperparameters tuning for english-assamese, english-mizo, english-khasi, and english-manipuri language pairs. cuni @xcite : the cuni team used back-translation @xcite for data augmentation, denoising, leveraging multilingual masked language modelling, and atulya-nits @xcite : this group used google colab, and trained the transformer model @xcite using a t4 gpu for the english-assamese, and english-manipuri language pairs. organizer: the shared task organizer used the opennmt toolkit @xcite and built bilstm-based nmt systems with hyperparameters tuning only on parallel data for the english-assamese, english-mizo, english-khasi, and english-manipuri language pairs. ## results and discussion we present results 4 for both directions of the four language pairs, namely, english-assamese in here, we have reported the evaluation scores of those teams who submitted system output and their associated papers. to evaluate quantitative results, standard evaluation metrics @xcite , namely, bleu (bilingual evaluation under study), ter (translation error rate) @xcite , ribes (rank-based intuitive bilingual evaluation score) @xcite , chrf (character 4 http://www2.statmt.org/wmt23/indic-mt-task.html n-gram f-score) @xcite and comet @xcite . moreover, we have hired linguistic experts who possess linguistic knowledge of the concerned language pair and randomly selected 20 sample sentences of primary submission type for manual evaluation (reported in team name organization bits-p birla institute of technology and science, pilani, india nits-cnlp national institute of technology, silchar, india onemt iiit-hyderabad, india sml lab iisc, bangalore, india nict-ai4b nict japan anvita centre for ai and robotics (cair), india muni-nlp masaryk university, czechia hv-nits national institute of technology, silchar, india irel-iiith iiit hyderabad india nvidia-india nvidia, india aimlnlp-iiti indian institute of technology, indore, india nlp_nith nit hamirpur, india transsion mt trannsion, china cnlp-iisc iisc, bangalore, india cuni charles university, czechia a3-108 ltrc, iiit hyderabad, india iol research transn, china slp-bv banasthali vidyapith, india iacs-lrilt indian assosciation for the cultivation of science, india nitr nit rourkela, india iit-nlp lab iit dharwad, india team siggymorph university of british columbia, canada lexical wizards kalinga institute of industrial technology, india junlp jadavpur university, india atulya-nits national institute of technology, silchar, india cfilt-iitb indian institute of technology, bombay, india cognitive lab-iiitm indian institute of information technology, manipur, india lrnmt-iiith iiit hyderabad, india guit-nlp gauhati university, india trddc tcs research, india hw-tsc huawei translate center, china ## conclusion we presented the results of the participating teams in the four language pairs translation task in terms of automatic and human evaluation metrics. we released a dataset, namely, indicne-corp1.0 in the shared task on low-resource indic language translation at the eighth conference on machine translation (wmt) 2023. the dataset comprises four low-resource languages, namely, assamese, mizo, khasi, and manipuri which belong to the northeastern region of india. in the future, we will include more northeastern indic language datasets in addition to increasing the existing dataset size.