First Attempt at Building Parallel Corpora for Machine Translation of N ortheast I ndia’s Very Low-Resource Languages (2023)

Abstract: This paper presents the creation of initial bilingual corpora for thirteen very low-resource languages of India, all from Northeast India. It also presents the results of initial translation efforts in these languages. It creates the first-ever parallel corpora for these languages and provides initial benchmark neural machine translation results for these languages. We intend to extend these corpora to include a large number of low-resource Indian languages and integrate the effort with our prior work with African and American-Indian languages to create corpora covering a large number of languages from across the world.

## introduction in the last few years, there have been significant advancements in deep learning approaches, such as the development of transformers @xcite . these advancements have greatly improved machine translation (mt), a core natural language processing (nlp) task. regarding coverage and translation quality, mt has shown remarkable improvements @xcite . most models and methods for high-resource languages do not perform well in low-resource settings. low-resource languages have also suffered from inadequate language technology designs @xcite @xcite @xcite @xcite . creating effective methods for natural language tasks is challenging, with extremely limited resources and little to no available data. the problem becomes worse without a parallel dataset for a vast number of world's languages @xcite @xcite . ethnologue 1 enumerates 7,618 extant human languages in the world, of which it estimates 3,072 1 www.ethnologue.com are endangered. india has 30 languages spoken by more than a million native speakers, 92 additional languages spoken by more than 10,000 people, and 1599 other languages. the terms language and dialect may have been conflated in census of india reports foot_0 . of these, 197 languages are endangered @xcite . the languages with which work in this paper are a small fraction of such languages. this paper makes a selection of low-resource languages of india, all from northeast india, with the intention of developing resources to facilitate neural machine translation. we choose northeast india because this area has a very high concentration of very low-resource and/or endangered languages, and the area being remote from mainland india has exacerbated scholarly neglect. ## related work dash (2020) reviewed the issues languages with a small number of speakers face in india. based on extensive research, the author surmised that the amount of published work on low-resource and endangered languages of india, especially in the context of technology and, in particular, natural language processing, is miniscule. @xcite provided a monolingual corpus of kangri (a language spoken by 1.1 million people in himachal pradesh and punjab) with 1.8m words, as well bitext hindi-kangri corpus of 27 thousand words. for nmt, they used a model that learns to map word embedding from one language to another in an unsupervised manner @xcite . only cursory details are given regarding the architecture of the neural model, although translation metrics are given in terms of bleu and meteor scores. the corpora are available on github foot_1 . acharya (2021) discussed a proposal for building a digital archive to collect and preserve textual, audio, and video documentation of twelve munda languages, a sub-family of austro-asiatic languages. of twenty munda languages, spoken by between just 20 to 7m people, unesco has identified twelve as endangered. the authors proposed to use advanced technologies like artificial intelligence in the design of the archive. however, it is not clear what has been achieved. to the authors' knowledge, there is no prior computational work of any kinda in the very lowresource and endangered languages of northeast india. ## languages under consideration the sino-tibetan language family includes more than 400 modern languages spoken in china, india, burma, and nepal. it is one of the most diverse language families in the world, with 1.4 billion speakers, including chinese, tibetan, and burmese. based on a phylogenetic study of 50 ancient and modern sino-tibetan languages, scholars have recently concluded that the sino-tibetan languages originated in north china around 7,200 years ago @xcite . various classification schemes have been proposed for the sino-tibetan family of languages @xcite @xcite . our discussion follows matisoff's classification that divides sino-tibetan languages into a number of sub-families at various levels. ## conclusions and future work this paper presents the first, albeit limited size, parallel corpus for 13 low-resource sino-tibetan indian languages paired with english and discusses the benchmark results for the translation of language pairs to/from low-resource indian languages from/to english. we evaluated the usability of the collected corpus by using transformer and finetuning multilingual translation model. from our results fine-tuning multilingual model outperforms transformer model trained from scratch in both translation directions. in the future, we aim to increase corpus sizes of these low-resource languages by extracting text from scanned documents if/where available and evaluate additional machine translation approaches to improve performance. we intend to increase the number of languages substantially by first incorporating all low-resource languages of india for which a bible translation exists. we also plan to find language communities in social media platforms such as facebook and attempt to gather additional bitext documents and evaluate the quality of translations with native speakers.