Revitalization of Indigenous Languages through Pre-processing and Neural Machine Translation: The case of I nuktitut (2020)

Abstract: Indigenous languages have been very challenging when dealing with NLP tasks and applications because of multiple reasons. These languages, in linguistic typology, are polysynthetic and highly inflected with rich morphophonemics and variable dialectal-dependent spellings; which affected studies on any NLP task in the recent years. Moreover, Indigenous languages have been considered as low-resource and/or endangered; which poses a great challenge for research related to Artificial Intelligence and its fields, such as NLP and machine learning. In this paper, we propose a study on the Inuktitut language through pre-processing and neural machine translation, in order to revitalize the language which belongs to the Inuit family, a type of polysynthetic languages spoken in Northern Canada. Our focus is concentrated on: (1) the preprocessing phase, and (2) applications on specific NLP tasks such as morphological analysis and neural machine translation, both for Indigenous languages of Canada. Our evaluations in the context of lowresource Inuktitut-English Neural Machine Translation, showed significant improvements of the proposed approach compared to the state-of-the-art.

## introduction north america and canada represent a highly complex linguistic regions, with numerous languages and great linguistic diversity. indigenous languages are spoken widely and are official languages in nunavut and the northwest territories. indigenous peoples are making efforts to revitalize and sustain their languages, although they face many difficulties. the case of inuktitut language in canada has recently been a good example to examine. this language belongs to the inuktut -the term inuktut is inclusive of all dialects used in nunavut -one of the official languages of nunavut, the largest territory of canada, which can be written in syllabics or in roman orthography, and regional variations use different special characters and spelling conventions @xcite . as inuktitut is an official language in nunavut, there exists some resources that are available in this language at a much larger scale than most other languages in the same family, notably a parallel corpus with english. the researchers have shown more interest in inuktut languages in nlp communities to respond to an increasing demand for indigenous language educational materials and technology. previous studies have demonstrated that the development of indigenous language technology faces many challenges on one hand, due to a high morpheme per word rate and the rich morphophonemics and variable dialectal-dependent spellings, a complex polysynthesis in linguistic typology, and on the other hand, due to the lack of orthographic standardization and a large dialect variation. digital text and voice data are limited. it poses many greater challenges for nlp to develop applications for all users @xcite . this research paper examines the case of inuktitut through experiments on the crucial phase as preprocessing and nlp task, to revitalize the language which belongs to the inuit languages family -the polysynthetic languages spoken in northern canada. we make our focus on (1) the preprocessing phase to build (2) nlp applications for indigenous languages, such as morphological analysis, parsing, named entities recognition and machine translation. this first step towards a multilingual nmt framework that will involve several endangered indigenous languages of canada, is essential, as the only parallel corpus freely available for research is the nunavut-english hansard corpus @xcite . the main contribution of our research is to revitalize these indigenous languages and help ancestral and other knowledge transmission vertically from the elders to the youth. the structure of the paper is described as follows: section 2 presents the state-of-the-art on mt involving indigenous languages. in section 3, we describe our proposed approach. then, in section 4, we present our experiments and evaluations. finally, in section 5, we present our conclusions and some perspectives for future research. farley (2012) developed a morphological analyzer for inuktitut, which makes use of a finite-state transducer and hand-crafted rules. @xcite presented an evaluation about the morphological analyzer for inuktitut, proposed by @xcite , and about alignment error rate with the use of the english-inuktitut nunavut hansard corpus. they reported the best experimental results, in terms of head approach which, in inuktitut, corresponds to the first one or two syllables of a token, with 79.70% precision, 92.20% recall and 17.60% alignment error rate. inspired by the uqailaut project of @xcite , @xcite applied a segmental recurrent neural network approach @xcite from the output of this morphological analyzer for inuktitut. the few studies that deal with machine translation while involving indigenous languages are related to the lack of parallel corpora @xcite . ## our proposed approach inspired by the work of @xcite , on the creation of the first inuktitut morphological analyzer based on the finite-state transducer method, we built a deep learning-based word segmentation tool for inuktitut. with the emergence of deep learning and the high computation of technology, neural networkbased approaches have shown their effectiveness when applied on word segmentation and enhanced with large-scale raw texts to pretrain embeddings. the neural network-based model, with these additional linguistic factors, can be able to deal with data sparseness or language ambiguity @xcite . following our previous research (le and sadat, 2020), we propose a two phases framework to build : (1) a bidirectional lstm word segmentation for inuktitut; and (2) an inuktitut-english nmt system (figure 1 ). in the first phase, the word segmentation task, considered as a sequence labeling task, is formally formulated as follows: given an input sequence, w and c represent all the word-based (bi-)character-based pretrained embeddings. in the input representation layer, the input sequence is vectorized based on word embeddings w . this vectorized sequence is concatenated with (bi)character-based pretrained embedding c, with the state w, c . then it is fed into a bidirectional lstm (long-short term memory) @xcite . a hidden feature layer h merges all input features x w , x c into a single vector with a k-dimension. the activation function, as an output function, is calculated in the output layer o, e.g. softmax. the second phase consists of building a nmt for inuktitut-english. here, we use the transformerbased encoder-decoder architecture @xcite . we apply our method to preprocess the inuktitut source language within an inuktitut-english nmt system. the masked language modeling methods of bert-based pretraining @xcite for nmt are time and resource consuming because they need to pretrain large models from scratch using large-scale data. instead of pre-training a bert model for inuktitut and english, we have intend to use the output of bert as context-aware embeddings of the nmt encoder. due to the lack of resources of inuktitut, we apply only the masked language model (mlm) concept @xcite to train our own monolingual bert-like model for inuktitut. for english, we will use pretrained models from huggingface foot_0 . the main goal aims to use pretrained bert-like model as embeddings in order to initialize the encoder of nmt model. the bert architecture can deal with two kinds of objective functions, namely, masked language modeling and next sentence prediction. in our context, we consider the masked language modeling objective of @xcite . in the training step, 15% words in a sentence are randomly masked and the mlm model is trained to predict them with their surrounding words. in other words, the mlm aims to reconstruct the original sentence sequence from noisy input because some words are masked with special tokens such as [mask] or [s] or [/s] for the beginning and the end of a sequence. for example, if the original sentence is "[/s] take a seat [/s]" and "[/s] [mask] a seat [mask]" is given as the input word sequence, then the system predicts that the original word for the first [mask] was "take" and the second [mask] was "[/s]"(figure 2 ). ## conclusion and perspective in this paper, we have presented how to leverage inuktitut-english neural machine translation with morphological word segmentation. we intend to apply our proposed approach in other indigenous language families, especially related to the inuit language family, to deal with nlp tasks. with the valuable collaboration of indigenous communities, we will be able to collect reliable data from the speakers of these indigenous languages. moreover, our nlp applications could help preserve ancestral knowledge and revitalize indigenous languages, heritage and culture with the transfer of knowledge from elders to the youth.