Enriching W ayúunaiki- S panish Neural Machine Translation with Linguistic Information (2023)

Abstract: We present the first neural machine translation system for the low-resource language pair Wayúunaiki–Spanish and explore strategies to inject linguistic knowledge into the model to improve translation quality. We explore a wide range of methods and combine complementary approaches. Results indicate that incorporating linguistic information through linguistically motivated subword segmentation, factored models, and pretrained embeddings helps the system to generate improved translations, with the segmentation contributing most. In order to evaluate translation quality in a general domain and go beyond the available religious domain data, we gather and make publicly available a new test set and supplementary material. Although translation quality as measured with automatic metrics is low, we hope these resources will facilitate and support further research on Wayúunaiki.

## introduction due to a lack of data (text or speech data), languages are digitally divided between high and lowresourced (lrl) @xcite . actually, the lowresource scenario has been identified as one of the main challenges in the field of natural language processing (nlp) @xcite . at the same time, research conducted and presented at major conferences often focuses on a few highly resourced languages, languages with similar characteristics, or a handful of well-studied languages @xcite . fortunately, research in lowresource settings and with lrls is slowely becoming quite popular in the nlp community, with a steadily growing body of work for the low-resource scenario @xcite . this does not imply that the division between low-and high-resourced nlp scenarios has been overcome. in fact, there are many open challenges for research on and with lrls. the majority of the world's 7000 languages are understudied and underresourced @xcite , due to the lack of research and resources. lrls face a lack of data quality and quantity, nlp tools, and engagement with native speakers of that language, which, if overcome, can support the conservation and preservation of those languages and their culture, preserving cultural and linguistic diversity. in this work, we aim at fostering research for wayúunaiki by providing data and pretrained neural machine translation (nmt) models. we present the first wayúunaiki-spanish nmt system, and explore different approaches to inject linguistic knowledge to improve translation quality. we aim at assisting the wayúu community, whose language is emerging from an endangered situation according to ethnologue. 1 even though the wayúu people are the most numerous indigenous people in colombia (departamento administrativo nacional de estadística, 2021), wayúunaiki is vulnerable, i.e. the language is spoken by children but only in certain, restricted domains, for instance at home. our research hypothesis in this work is that the injection of linguistic knowledge will increase the translation quality for the language pair wayúunaki-spanish. we enrich the data to represent implicit linguistic information (e.g., linguistically motivated subword segmentation, annotating pos tag factors, and pretrained embeddings) as, if insufficient amounts of training data is available, linguistic information may help the model identify patterns present in the text, which may alleviate the data sparsity problem. we build on and extend previous work on nmt for lrls by @xcite and @xcite . we combine complementary approaches to maximize improvements. we find that while linguistically motivated subword segmentation helps, factored models and pretrained embeddings lead to a performance degradation due to data sparsity and low quality annotations. while the results of this work do not provide good quality translation models yet, we expect to contribute to the development of nmt systems for lrls and to inspire further research. we integrate our best-performing system for wayúunaiki to spanish into the document translation interface transins foot_1 @xcite for public use. our collected supplementary material, the new general domain test data set, as well as code are also publicly available. 3 2 related work various ways of incorporating linguistic knowledge into nmt systems have been explored. these include the addition of (linguistic) factors (e.g., sennrich and @xcite , @xcite , manzanares, 2020), or using different subword segmentation techniques (e.g., @xcite , @xcite with the aim of improving translation quality. improvements are possible, especially in lrl scenarios (e.g., @xcite , morphologically rich languages (e.g., @xcite , and for out-of-domain texts (e.g., @xcite . subword segmentation is essential in nmt since it eases the out-of-vocabulary (oov) problem and allows training smaller models @xcite . subword units offer a representation, that builds a bridge between word and characterlevel, based on the statistical properties of the text. a good choice of subword units will offer a good balance between the vocabulary size, the size of the model and therefore the decoding efficiency. data-driven, unsupervised subword segmentation is a statistically-informed process that incorporates implicit linguistic knowledge present in the text, like statistical patterns that present regularities of encountered word forms. this approach is limited to the data used during training the segmentation model, such that text variations (e.g., inconsistent orthography or out-of domain context) might result in segmentation variations and oversegmentation @xcite . the byte-pair-encoding (bpe) algorithm @xcite is a widely used, unsupervised approach for subword segmentation. bpe merges the most fre-quent pairs of characters in a corpus to create a new subunit, and repeats the process until the desired number of merge operations are performed. with bpe, common words form a single unit while rare words are split into subunits. the first application in mt by @xcite lead to a strong improvement in performance. further approaches include sentencepiece @xcite , a tokeniser that implements both bpe and unigram language model (lm) @xcite . in @xcite subword segmentation is combined with a regularization method, offering a robust alternative to the deterministic bpe. for the segmentation technique by @xcite , an initial subword set is pruned, according to the contribution of each subword to the unigram lm @xcite . another alternative for creating more segmentation variety in the training data is the regularization method particularly for bpe called bpe-dropout @xcite . semi-supervised segmentation techniques incorporate and exploit linguistically labeled training data to guide the segmentation process. linguistic annotation can help to learn the correct segmentation rules, especially in low quantity and quality data scenarios @xcite . @xcite is a morphologically guided algorithm, that incorporates linguistic knowledge without requiring any morphological rules. nonetheless, a list of affixes is essential during the construction of the segmenter. in comparison to the bpe algorithm, subwords that include positional information of a word are extracted in form of prefixes, roots, and postfixes. this subword segmentation algorithm has been shown to improve translation quality, measured with bleu, in comparison to other systems, in which unsupervised algorithms were applied @xcite . the algorithm is not thought to be used as a morphological segmentation tool, even though it produces text that resembles morphologically segmented text. moreover, it avoids over-segmentation by sometimes only partially performing the morphological splitting with the motivation that too many subwords would reduce the translation quality @xcite . ## language description wayúunaiki is the native language spoken by a minority (compared to spanish) in the wayúu community, located in the caribbean region, connecting colombia and venezuela. more than half a million people of this bi-national community speak this lrl. the wayúu community is the most numerous indigenous community in colombia (departamento administrativo nacional de estadística, 2021). there are 380,460 wayúus in colombia foot_4 and about 415,500 wayúus in venezuela @xcite . wayúunaiki belongs linguistically to the arawak languages. this language family flourished among ancient, indigenous nations in south america and consists of polysynthetic, mainly head-marking languages with different degrees of agglutination (méndez-rivera, 2020). spanish, the highresourced language spoken in the same countries, is a fusional, inflected language with a flexible syntactic order. the preferred pattern is subject + verb + object (svo), while wayúunaiki has a vso order. both languages have their own phonological system and do not share the same alphabet: spanish has 22 consonants and 5 vowels in its phonological repertoire, while wayúunaiki has 16 consonant and 12 vowel phonemes -6 vowel pairs of long and short ones @xcite ). an inconsistent writing system for the wayúu language, due to the two main "official" orthographic systems, in combination with a very small amount of written material in wayúunaiki, make the orthographic situation challenging @xcite . 6 4 data collection and preprocessing parallel corpora. we use the only online parallel corpus for wayúunaiki and spanish available in the tatoeba mt challenge, version v2021-08-07 @xcite . the bitext is a subpart of the no longer available jw300, a parallel corpus from @xcite with religious-themed data, addressing a wider range of topics including bible psalms. 7 the wayúunaiki part of the bitext follows the official writing norm aliv (alfabeto de lenguas indígenas de venezuela, alphabet of indigenous languages of venezuela). the corpus consists of ∼43k sentence pairs, which we divided into a train, development, and test set. in order to create a general domain parallel data set and assess the generalizability of the translation systems, we collected data from spanish-wayúunaiki dictionaries and illustrative grammar booklets for non-wayúunaiki speakers to learn the language. monolingual corpora the monolingual corpus is used in our work combined with the monolingual parts of the parallel corpus to train word embeddings. supplementary material some of our experiments require supplementary information in the form of linguistic annotations, or dictionaries. we extracted morphological analyses of verb conjugations in wayúunaiki from the work of @xcite to guide the semi-supervised training of the segmentation models (prefix-root-postfix-encoding and flatcat). for this, the morph categories prefix, stem, and suffix were manually annotated. an example file is listed in appendix a and we make all files available online. 10 we perform a similar morphological annotation with spanish samples taken from lecture slides from doctor lluís simarro lacabra (2014), an educational institution. preprocessing we split the monolingual text into sentences and tokens using the nltk tokenizer. since there is no tokenizer for wayúunaiki, we use regular expressions (re). the character ' in wayúunaiki, which in the latin alphabet represents the glottal stop consonant [p] known as "saltillo", little skip, had to be stripped from additional white spaces. for simplification, all possible saltillos ( ' ′ ' ´) were mapped to the ' character in the parallel data sets. likewise, quotations ( » « " " ) were normalized to ". bible verses number references were detected with res and removed. enumerations with brackets, numbers with punctuation at the beginning of the sentence, and urls were also removed. we train a truecaser with moses scripts @xcite for each language on the parallel data and applied them to all data sets accordingly. ## nmt systems all our models are based on a transformer architecture @xcite and developed with marian v1.11.0 @xcite . ## results and discussion we report the translation scores for wayúunaiki to spanish in tables 4 (religious domain) and 5 (general domain) for each method with the best system per metric boldfaced. in pretrained embeddings alone do not significantly improve the results (base+emb, subw-bpe+emb), although they have been shown to provide a better representation of less frequent concepts in lrls @xcite . @xcite showed that pretrained embeddings seem to be effective for not-too-distant translation pairs. this may well be the reason for our lack of improvement, wayúunaiki and spanish are very distant, but we conjecture that the most important problem we face is the lack of sufficient data to train wayúu embeddings: the monolingual wayúu corpus we use is almost equivalent to the size of the parallel corpus. still the results of @xcite indicate that pretrained embeddings seem to introduce semantic and syntactic information of words improving translations even for distant translation pairs: systems are able to capture overall basic language characteristics and generate more grammatically well-formed sentences. @xcite indicate that for very little but sufficient training data, that allows training the system, using pretrained word embeddings from @xcite are most effective. their usage of pretrained embeddings by @xcite make comparison with our results very difficult, as such embeddings are trained on billions of tokens. in an out-of-domain evaluation their model outperformed all of their systems trained with different segmentation methods (e.g., bpe, unigram lm, prpe). ## conclusion and future work in this work we applied various unsupervised and semisupervised subword segmentation methods to enrich the data used to train a transformer-based nmt model with linguistic information. additionally, we extended the architecture of the standard subw-bpe model by adding linguistic information in the form of pos tag factors and/or supplying the system with pretrained embeddings. in line with previous research on indigenous lrl pairs that include spanish, we observed that the addition of subword information is crucial to improve translation quality (e.g., @xcite , @xcite , @xcite . in particular, the indigenous languages of america, which are mostly characterized by a rich morphology, and part of agglutinative and polysynthetic languages, benefit from approaches that consider the lrl's morphology and apply subword segmentation techniques that are suitable for the language pair. in contrast, we did not achieve any improvement with factors and pretrained embeddings. the lack of resources, in terms of data and annotation coverage, is the likely cause for the low performance of these models. our next steps are focused on investigating the effectiveness of injecting linguistic knowledge for the wayúu language by exploring datasets without repetitive sequences and less sparse and noisy annotations. to do this, more sophisticated approaches to obtain implicit linguistic knowledge from lrl text, such as introducing linguistic information also on the target side in the form of pos-tag or lemma factors are possible. problems related to the lack of resources for factored training could in principle be overcome by applying a linguistically inspired subword segmentation technique, for instance, morfessor's flatcat. by splitting a word into its subwords, chances of determining the stem are higher, if the segmentation into subwords representing stems and suffixes is both accurate and consistent. given the stem, the word can be annotated with its pos tag from the linguistic knowledge-based vocabulary. we note that this is limited to languages without infixation and would work only for words without assimilatory processes between affixes and stem. still, it presents a possible approach to obtain labeled data. besides enriching the data with linguistic information, our observations on word repetitions and hallucinations indicate that additional cleaning, filtering of unaligned source and target translations, and orthographic normalization could significantly enhance data quality and hence translation performance. we believe that injecting linguistic information, especially for lrl pairs can alleviate the data sparsity problem and aid the models with the annotation of implicit linguistic knowledge present in the data. by enriching the data to represent such information present in the text (e.g., annotating pos tags), a model can better identify patterns inherent in the data. still, choosing between the different approaches and techniques requires taking into account the nature of the lrl pair and the available resources, particularly supported nmt tools and data sets.