I nd T 5: A Text-to-Text Transformer for 10 Indigenous Languages (2021)

Abstract: Transformer language models have become fundamental components of NLP based pipelines. Although several Transformer have been introduced to serve many languages, there is a shortage of models pre-trained for low-resource and Indigenous languages in particular. In this work, we introduce IndT5, the first Transformer language model for Indigenous languages. To train IndT5, we build IndCorpus, a new corpus for 10 Indigenous languages and Spanish. We also present the application of IndT5 to machine translation by investigating different approaches to translate between Spanish and the Indigenous languages as part of our contribution to the AmericasNLP 2021 Shared Task on Open Machine Translation. IndT5 and IndCorpus are publicly available for research.

## introduction indigenous languages are starting to attract attention in the field of natural language processing (nlp), with the number of related publications growing in recent years @xcite . in spite of this interest, there remains a multitude of challenges for handling indigenous languages. complexity of the morphological systems of some of these languages and lack of standard orthography for writing them are among these challenges @xcite . the most fundamental issue facing nlp efforts, however, remains the lack of digital textual data that can be exploited for systems development. in this work, we describe a scenario usually faced when trying to develop nlp systems for indigenous languages and we focus on machine translation (mt). we adopt a neural machine translation approach (nmt) @xcite as our method. we show that, in spite of its recent success on many contexts, nmt still struggles in very low-resource settings involving indigenous languages. this is due to the core difficulty of lack of parallel textual data, but also even monolingual data. although our main goal in this work in particular is to develop translation models from spanish to several indigenous languages of the americas, we adopt a transfer learning approach where we offer resources that can be exploited for other downstream tasks. namely, we build a dataset for ten indigenous languages and spanish which we refer to as indcorpus. figure 1 and indt5, treats every text nlp problem as a "textto-text" problem, i.e. taking text as input and producing new text as output. we apply indt5 to the mt task as a way to transfer knowledge acquired by the model to this particular context. our experiments show the utility of our new language model and the dataset it exploits for the downstream indigenous mt task but that very large space for improvement still exists. the rest of the paper is organized as follows: in section 2, we introduce recent mt work in lowresource and indigenous languages settings. in section 3, we describe how we develop our new language model for ten indigenous languages. in section 4, we describe our nmt models. we conclude in section 5. ## indt5 we train an indigenous language model adopting the unified and flexible text-to-text transfer transformer (t5) approach @xcite . t5 treats every text-based language task as a "textto-text" problem, taking text format as input and producing new text format as output. t5 is essentially an encoder-decoder transformer @xcite , with the encoder and decoder similar in configuration and size to a bert base @xcite but with some architectural modifica-tions. modifications include applying a normalization layer before a sub-block and adding a pre-norm (i.e., initial input to the sub-block output). we call our resulting model indt5. we now describe our dataset, vocabulary, and pre-training method for developing indt5. ## conclusion in this work, we introduced a new transformer language model (indt5) and a dataset (indcorpus) for ten indigenous languages and spanish. we applied indt5 to the mt task on eight languages pairs as part of our submission to the americasnlp 2021 shared task. while indt5 helps improve translation, the task remains hard due to absence of parallel as well as monolingual data. in the future, we plan to integrate statistical mt methods to augment our data as well as investigate best hyperparameters for our neural models.