Parallel Corpus for Indigenous Language Translation: S panish-Mazatec and S panish- M ixtec (2023)

Abstract: In this paper, we present a parallel Spanish- Mazatec and Spanish-Mixtec corpus for machine translation (MT) tasks, where Mazatec and Mixtec are two indigenous Mexican languages. We evaluated the usability of the collected corpus using three different approaches: transformer, transfer learning, and fine-tuning pre-trained multilingual MT models. Fine-tuning the Facebook m2m100-48 model outperformed the other approaches, with BLEU scores of 12.09 and 22.25 for Mazatec-Spanish and Spanish-Mazatec translations, respectively, and 16.75 and 22.15 for Mixtec-Spanish and Spanish-Mixtec translations, respectively. The results indicate that translation performance is influenced by the dataset size (9,799 sentences in Mazatec and 13,235 sentences in Mixtec) and is more effective when indigenous languages are used as target languages. The findings emphasize the importance of creating parallel corpora for indigenous languages and fine-tuning models for low-resource translation tasks. Future research will investigate zero-shot and few-shot learning approaches to further improve translation performance in low-resource settings.

## introduction natural language processing (nlp), a sub-field of artificial intelligence (ai), has been attracting a lot of attention in terms of research and development as a result of the surge in the number of applications it has in a variety of different industries @xcite . machine translation (mt), sentiment or opinion analysis, pos tagging, question classification (qc) and answering (qa), chunking, named entity recognition (ner), emotion detection, and semantic role labeling are currently highly researched areas in various high-resource languages @xcite . the domain of machine translation (mt) is advancing at a rapid pace due to the growing prevalence of computational tasks and the expanding global reach of the internet, which caters to diverse, multilingual communities @xcite . mt systems have demonstrated remarkable translation outcomes for language pairs that possess abundant resources, such as english-spanish, english-french, english-russian, and english-portuguese. however, in scenarios with limited or no resources, mt systems encounter difficulties due to the primary obstacle of inadequate training data for certain languages @xcite @xcite . low-resource languages have been suffering from a lack of new language technology designs. when the resources are limited and only a small amount of unlabeled data is available, it is very hard to reach a true breakthrough in creating powerful novel methods for language applications @xcite , the problem becomes worse if there is no parallel dataset for certain languages. mexico is a multicultural and multilingual country with 68 officially recognized indigenous languages, 238 variants, and spanish, a widely used language spoken by 90 percent of the population @xcite . few language technologies have been developed for indigenous languages spoken in northern and southern america; moreover, many indigenous languages spoken in the americas face a risk of extinction @xcite . indigenous language speakers often experience feelings of shame or reluctance to use their native languages, primarily due to limited opportunities for application in the presence of pervasive, dominant majority languages @xcite . this phenomenon can be attributed to social and cultural pressures that prioritize the use of majority languages over minority languages, thereby marginalizing indigenous linguistic communities and undermining the value of their linguistic heritage @xcite . in this paper, we introduce the first parallel corpus for machine translation tasks for two indigenous languages that are spoken in mexico and benchmark experimental results. the contributions of our work are the following: the rest of the paper is organized as follows: section 2 describes previous research related to this study, section 3 describes the properties of mazatec and mixtec languages, section 4 describes the statistics of the collected dataset, section 5 describes models used for baseline experiments and their results, and section 6 describes the conclusion of the paper. ## related works due to an increase in the enormous amount of data for different languages, machine translation is currently one of the most researched areas in nlp and has shown promising results in high-resource languages @xcite . there are different mt approaches that have been used by different researchers, neural machine translation (nmt)is one of the current state-of-the-art approaches trained on huge datasets containing sentences in a source language and their equivalent target language translations @xcite . basically, nmt takes advantage of huge translation memories with hundreds of thousands or even millions of translation units @xcite . however, nmt for lowresource languages still under-performs due to the scarcity of parallel datasets @xcite . many researchers explored different approaches to solving low-resource machine translation problems. @xcite proposed a transfer learning method to improve the mt performance of low-resource languages. the authors first train a high-resource language pair (the parent model), then transfer some of the learned parameters to the low-resource pair (the child model) to initialize and constrain training. the data augmentation approach proposed by @xcite , targets low-frequency words by generating new sentence pairs containing rare words in new, synthetically created contexts. @xcite proposed using high-resource language resources to improve mt performance for low-resource languages without requiring any parallel data. copying monolingual data of the target language is proposed by @xcite to improve the performance of low-resource mt. tonja et al. (2023b) proposed the use of source-side monolingual data as another way of improving low-resource mt performance. transfer learning method, where one first trains a "parent" model for a high-resource language pair and then continues training on a lowresource pair only by replacing the training corpus was proposed by @xcite . mixing low-resource language resources during training, as proposed by @xcite showed an improvement in mt performance for low-resource languages. there have been promising research works done for indigenous languages; feldman and coto-solano (2020) presented an nmt model and a dataset for the bribri chibchan language for bribri-spanish translation. @xcite compiled americasnli, a natural language inference dataset covering 10 indigenous languages of the americas. they conducted experiments with pre-trained models, exploring zero-shot learning in combination with model adaptation. @xcite proposed the first multilingual translation models for four languages spoken in peru: aymara, ashaninka, quechua, and shipibo-konibo, providing both many-to-spanish and spanish-to-many models, outperformed pairwise baselines. @xcite presented a low-resource mt system that improves translation accuracy using crosslingual language model pre-training. the authors used an mbart implementation of fairseq to pretrain on a large set of monolingual data from a diverse set of high-resource languages before finetuning on 10 low-resource indigenous american languages: aymara, bribri, asháninka, guaraní, wixarika, náhuatl, hñähñu, quechua, shipibo-konibo, and rarámuri. on average, their proposed system achieved bleu scores that were 1.64 higher and chrf scores that were 0.0749 higher than the baseline. nagoudi et al. (2021) introduced indt5, the first transformer language model for 10 indigenous american languages: aymara, bribri, asháninka, guaraní, wixarika, náhuatl, hñähñu, quechua, shipibo-konibo, and rarámuri. to train indt5, they built indcorpus-a new dataset for ten indigenous languages and spanish. ## baseline experiment and discussion in this section, we discuss the models used for the baseline experiment, the hyper-parameter used, the benchmark results, and the discussion. we used three approaches to evaluate the usability of the collected corpus. these are :- the key innovation of the transformer architecture is the attention mechanism, which allows the network to selectively focus on different parts of the input sequence when making predictions. this is in contrast to traditional recurrent neural networks (rnns), which process input sequentially and are prone to the vanishing gradient problem. in the transformer architecture, the input sequence is processed in parallel by multiple layers of self-attention and feed-forward neural networks. each layer can be thought of as a "block" that takes the output of the previous layer as input and applies its own set of transformations to it. the self-attention mechanism allows the network to weigh the importance of each element in the input sequence when making predictions, while the feed-forward networks help to capture nonlinear relationships between the elements. currently, transformers are state-of-the-art approaches and are widely used in nlp tasks such as mt, text summarization, sentiment analysis, etc. we used the base transformer configuration as described in @xcite work. transfer learning can be especially useful in mt because training a high-quality mt system from scratch requires a large amount of data and computational resources, which may not be available for all language pairs or domains. by leveraging pre-trained models, transfer learning allows mt systems to achieve high performance with fewer data and fewer resources. for our baseline experiments, we used english-spanish as parent model with two (opus-mt-es-en 1 and opus-mt-tc-big-enes 2 ) pre-trained models available from hugging face 3 trained for english-spanish on the opus dataset @xcite ) by helsinki-nlp group. hyper-parameters -for the transformer approach we tokenized the source and target parallel sentences into subword tokens using byte pair encoding (bpe) @xcite . the bpe representation was chosen in order to remove vocabulary overlap during dataset combinations. for other approaches we applied the tokenizer of each model, ## conclusion in this paper, we presented a parallel corpus for two indigenous mexican languages (mazatec (maq) and mixtec (xtn)) for machine translation tasks and evaluate the usability of the collected corpus using three different approaches. from the approaches, fine-tuning multilingual pre-trained mt models outperformed the rest of the experiments; facebook's m2m100-48 outperformed all other models with bleu scores of 12.09 and 22.25 for maq-spa and spa-maq, respectively, and 16.75 and 22.15 for xtn-spa and spa-xtn, respectively. we noticed from the experimental results that the dataset size has less impact when using indigenous languages as a target than the source. this observation highlights the potential benefits of focusing on developing and fine-tuning models specifically designed for translation tasks involving low-resource languages. moreover, it underscores the value of creating and employing parallel corpora tailored to indigenous languages, as these resources can significantly improve machine translation performance, particularly when used in conjunction with advanced multilingual pre-trained models. our bleu results for mizatec and miztec to spanish translation were very low on the best configuration to have any usability in real-life applications, but the translation in the opposite direction demonstrated bleu scores above 22 facilitating uses, for example in government apps to present hints to mixtec and mazatec native speakers who have a low level of spanish comprehension, in the government web pages. this could significantly improve the usefulness of the native language of the speakers, thus promoting communication of the language and its preservation. in future research, we plan to investigate the efficacy of advanced techniques, including zero-shot and few-shot learning, for low-resource languages in the context of limited parallel datasets. these methodologies hold promise for effectively leveraging sparse data available in low-resource settings, as they capitalize on pre-existing knowledge from related tasks or languages without requiring extensive fine-tuning or additional annotated data. by exploring these approaches, we aim to uncover potential benefits and improvements in the machine translation performance of low-resource languages, thus contributing to developing more robust and accurate translation systems for underrepresented linguistic communities.