Word-level prediction in P lains C ree: First steps (2024)

Abstract: Plains Cree (nêhiyawêwin) is a morphologically complex and predominantly prefixing language. The combinatory potential of inflectional and derivational/lexical prefixes and verb stems in Plains Cree makes it challenging for traditional auto-completion (or word suggestion) approaches to handle. The lack of a large corpus of Plains Cree also complicates the situation. This study attempts to investigate how well a BiLSTM model trained on a small Cree corpus can handle a word suggestion task. Moreover, this study evaluates whether the use of semantically and morphosyntactically refined Word2Vec embeddings can improve the overall accuracy and quality of BiLSTM suggestions. The results show that some models trained with the refined vectors provide semantically and morphosyntactically better suggestions. They are also more accurate in predictions of content words. The model trained with the non-refined vectors, in contrast, was better at predicting conjunctions, particles, and other non-inflecting words. The models trained with different refined vector combinations provide the expected next word among top-10 predictions in 36.73 to 37.88% of cases (depending on the model).

## introduction auto-complete systems and predictive text input have become integral components of our daily interactions with our devices and digital platforms. these applications heavily rely on robust language models capable of accurately predicting the next word in a given sequence of text. while substantial progress has been made in developing efficient language models for major languages, the challenges persist for low-resource languages where scarcity of training data poses a significant obstacle. this challenge is especially found for indigenous languages that are often also morphologically rich. with advances in the nlp and machine learning fields, small training datasets have become less of a problem; however, the handling of the morphological complexity still presents a challenge. @xcite approached this problem with the development of an interactive word-completion system for kunwingku (an indigenous language spoken in northern australia) based on a finite state recognizer which included most morphology for some 500 verbs. their tool suggests a completion up to the next morpheme boundary and helps to avoid the so-called "combinatorial explosion of possible words" typical for the prefixing polysynthetic languages. @xcite further successfully extend this method to plains cree, with a full-fledged model including all parts of speech, covering most inflectional morphology for the inflecting verbs and nouns, and based on a lexicon of well over 20k lexemes. the tool is based on a finite state morphosyntactic analyzer of plains cree (nêhiyawêwin, an indigenous language spoken mainly in on the western canadian plains) @xcite . it uses corpus-based information about cree prefixes to predict the most probable and common next morpheme in a word (based on a small corpus of some 150k cree words). while the results were perceived as surprisingly good, given the small size of the corpus, there remained yet quite many valid optional completions, since their tool did not make use of preceding lexical or morphosyntactic context. similarly, with @xcite , plains cree interactive word completion could be used by non-fluent cree speakers and learners who may struggle to build word forms. nevertheless, as the successful use of the model requires a broad knowledge of the language and its word formation, in order to be able to choose the completion appropriate to the context, they considered that the compilation model might be most useful for fluent speakers. additionally, the model is helpful to fluent speakers who have difficulties with diacritics for vowel length and other aspects of spelling, in support of which they also included a spelling correction component. the present research draws inspiration from these pioneering works of @xcite and @xcite , and seeks to continue the experiments in the field of word completion for plains cree. the present study, however, aims to investigate the feasibility of a complete word prediction and seeks to provide fluent cree speakers with morphosyntactically and contextually appropriate, if not accurate, word suggestions which can potentially speed up the typing process. to achieve this, we train a bidirectional long short-term memory (bilstm) model to predict the next word in a sequence. lstms, a type of recurrent neural network (rnn), have demonstrated remarkable success in capturing contextual dependencies in sequential data, making them a compelling choice for natural language generation tasks @xcite . moreover, the fact that several studies working with low-resource agglutinative and polysynthetic languages used lstms for word prediction task @xcite , makes it a compelling choice for the plains cree case. to improve the model performance, we also train word2vec embeddings @xcite for words in the plains cree corpus (see section 2.2). additionally, in this study, we explore the effect of vector augmentation-based on the words' morphosyntactic analyses, wordnet semantic classes (if applicable) @xcite , and lemmas-on the overall model accuracy and quality of word suggestions. the paper is structured as follows. the data used in this paper and data preprocessing are described in section 2. the word2vec vectors training and refinement, and bilstm model training are described in section 3. section 4 presents the results, and section 5 presents their discussion. possible directions for further research are examined in section 6. ## discussion and conclusions although testing of the models shows that the overall accuracy is higher for the simple bilstm, we argue that these results need further analysis and discussion before we can come to the final conclusion about vector refinement efficiency for plains cree word prediction. in order to quantify the results of our qualitative observations, we did two additional tests on the out-of-fold prediction results. first, we analyzed how semantically close were the predictions to the expected word. second, we measured the morphosyntactic similarity of the predictions and the expected words. for the first test, we measured a wu-palmer similarity between the wordnet classes of the predictions and the target words with the nltk wordnet package. the wu-palmer similarity value repre-input: ...cêskwa! itwêw awa sihkihp. êkotê isi kapâw. miyosiyiwa ôhi _ eng: wait! s/he says this is a waterhen. towards there, s/he goes ashore. someone is beautiful, this is_' no. full bilstm predictions bilstm predictions 1 ka-preverb kâ-preverb 2 oskinîkiskwêwa 'another young woman' iskwêwa 'another woman' 3 iskwêwa 'another woman' oskinîkiskwêwa 'another young woman' 4 oskinîkiwa 'another young man' ê-preverb 5 wâkayôsa 'another black bear' oskinîkiwa 'another young man' 6 nâpêwa 'another man' nâpêwa 'another man' 7 kisêyiniwa 'another old man' another wâkayôsa 'another black bear' 8 nâpêsisa 'another boy' mostoswa 'another cow' 9 okimâwa 'another chief' kisêyiniwa 'another old man' 10 nôtokêsiwa 'another old woman' okimâwa 'another chief' for the second test, we calculated the jaccard coefficient index for the morphosyntactic analyses of target words and predicted words. this comparison intended to show how well the suggested words were able to fit the morphosyntactic structure of the sentence. the third column of the results of the tests showed that word+analysis and word+analysis+lemma refined models provided contextually and grammatically better suggestions in comparison to other models. an average suggestion of the word+analysis model was 42.5% grammatically and 38.22% semantically similar to a target word. however, the difference is too small to claim with certainty that the refined vectors significantly improved the quality of predictions. although vector refinement does not provide a substantial prediction improvement, the main experiment and the additional tests indicate that the morphosyntactic information about words con-tributed the most to the refined models' accuracy and quality of suggestions. probably, this result is due to the lower number of words lacking morphosyntactic analysis (in comparison to words lacking wordnet class). the lemma information does not seem to contribute to the overall accuracy of prediction. nevertheless, it seems to improve the overall quality of suggestions. the information about neighbouring wordnet classes did not improve the accuracy or quality of word prediction. however, it is most likely related to the high number of words in the dataset that were not assigned a wordnet class yet or do not have a wordnet class. the analysis of the models trained with refined vectors and the regular model revealed a disadvantage of embeddings refinement in the present settings. although refined vectors contributed to the slight prediction quality improvement for the words that had all the extra information like morphosyntactic analysis, wordnet class, and lemma, they did not provide an adequate representation for the words lacking some or all of this extra information. the vector refinement function did not update its vectors based on the additional information. consequently, after refinement, they could appear further from the words they originally co-occurred with, because their 'neighbours' vectors were updated. this highlights the necessity for a better refinement approach in further studies. in this study we used simple vector averaging, but in the follow-up studies the more sophisticated approaches like those proposed by @xcite and @xcite should be explored. to interpret the results of this study, it is also important to keep in mind that the model was tested on a small chunk of the corpus. our corpus, in general, has a large portion that comes from transcribed plains cree narratives and fiction stories. transcribed narratives often have more filler words, while fictional stories often have rare literary words. both significantly differ from the writing we use on a day-to-day basis (texting, search queries, etc.). in some cases, the preceding context may have many words without semantic class and that makes predictions of the following words very tricky. thus, we are sure that under the present circumstances, the experiment with refined bilstm models training yielded promising results. in future, we want to experiment with more standardized texts for training and testing and explore the possibility of excluding filler words for the word2vec training. more-over, we believe that the predictions' quality can be also improved by reducing the number of nonanalyzed words with unknown wordnet classes in our corpus. further improvements in the training dataset can allow the model to learn more about contextual neighbours of each word, wordnet class and morphosyntactic analysis. to conclude, this research lays the groundwork for a future predictive text model for plains cree. it shows that full-word prediction is not impossible for plain cree, and with certain improvements and modifications, can reach higher accuracy levels. this study also explores the use of augmented word embeddings in data scarcity cases; however, the efficiency of this method requires further analysis with a fuller dataset. potentially, use of the model in tandem with other rule-based tools and resources developed for plains cree, such as morphosyntactic analyzer @xcite , constraint grammar parser @xcite , or weighted plains cree morpheme combinations @xcite , can lead to more accurate results. naturally, significant improvements are required before speakers and learners can use this tool.