Predictive Text for Agglutinative and Polysynthetic Languages (2022)

Abstract: This paper presents a set of experiments in the area of morphological modelling and prediction. We test whether morphological segmentation can compete against statistical segmentation in the tasks of language modelling and predictive text entry for two under-resourced and indigenous languages, K’iche’ and Chukchi. We use different segmentation methods — both statistical and morphological — to make datasets that are used to train models of different types: single-way segmented, which are trained using data from one segmenter; two-way segmented, which are trained using concatenated data from two segmenters; and finetuned, which are trained on two datasets from different segmenters. We compute word and character level perplexities and find that single-way segmented models trained on morphologically segmented data show the highest performance. Finally, we evaluate the language models on the task of predictive text entry using gold standard data and measure the average number of clicks per character and keystroke savings rate. We find that the models trained on morphologically segmented data show better scores, although with substantial room for improvement. At last, we propose the usage of morphological segmentation in order to improve the end-user experience while using predictive text and we plan on testing this assumption by doing end-user evaluation.

## introduction nowadays text prediction is widely used in different applications such as autocomplete tools, smart keyboards, etc. the used language models are limited by resources, so they can store only the top-n highest frequency words, which may work well with analytic languages, but when it comes to the synthetic languages the out-of-vocabulary (oov) problem becomes more and more noticeable. in order to deal with this problem, words are usually segmented in constituent parts, so that more of them can be saved in the model vocabulary. segmentation is almost always done using statistical methods, such as bpe @xcite . in this paper, we test whether morphological segmentation can improve language modelling and whether it can compete against statistical segmentation methods in predictive text entry task. the reason to suggest morphological segmentation is that we want text prediction to be both effective and ergonomic. by ergonomic we mean that predictions should be linguistically sound and intelligible for the end user. for example, imagine an english word antidisestablishmentarianism. an ergonomic segmentation will split the word into its constituent morphs @xcite , or an alternative @xcite . an unergonomic segmentation might be @xcite or @xcite . one of the issues with many current methods is that while they can produce segments that are meaningful units, in many cases the segments are not linguistically meaningful. we argue that for the task of predictive text entry producing non-linguistic units creates more cognitive load and so will result in slower text entry than predicting the same amount (or a greater number of) linguistic units. the remainder of the paper is laid out as follows: in section 2 we overview the languages we experiment on, in section 3 we discuss the works that were an inspiration for this paper, in section 4 we describe the experiments we are doing, in section 5 we review the used segmentation methods, in section 6 we provide results of language modelling, in section 7 we speak about language modelling evaluation task, in section 8 we discuss our thoughts on the results, in section 9 we announce the planned future experiments. examples in this paper will be mostly given in k'iche', chukchi and english. english examples, while english being neither an agglutinative or polysynthetic language, are given in order for the reader to better understand the examples. ## languages we perform the experiments using two languages: k'iche' (iso-639: quc), a mayan language of guatemala that is of the agglutinating type, and chukchi (iso-639: ckt), a chukotko-kamchatkan language of siberia of the polysynthetic type. both of these types are characterised by words consisting of a large number of individual morphs, surface representations of morphemes. the following examples in k'iche' (1) and chukchi (2) demonstrate this tendency. 1 (1) x-in-e'-ki-k'am-a' cp-b1sg-mov-a3pl-receive-dep 'they went to take me' both languages exhibit polypersonal agreement (both the subject and object arguments of transitive verbs are encoded on the verb), and chukchi, in addition, exhibits noun incorporation. as it can be seen in example 2, the object манэ /mane/ 'money' is incorporated, rendering intransitive the transitive root ванԓя /wanɬa/ 'ask'. ## related work being one of the latest works on language modelling of indigenous languages, @xcite proposed the usage of morphological segmentation in order to improve metrics of language modelling. the authors compared different segmentation methods, such as single words, dividing into characters, bpe, morfessor, finite-state transducers (fst). unfortunately, the authors could not do the end-task evaluation of the trained models but suggested doing predictive text as evaluation. @xcite , devoted to mi'kmaq language modelling evaluation, gave us ideas on how to approach the language modelling task. mi'kmaq (iso-639: mic), an eastern algonquian low-resourse polysynthetic language, is spoken primarily in eastern canada and has around 8700 speakers. not only did the authors work with indigenous language, but they also did the keystroke savings evaluation, which is pretty similar to the predictive text evaluation described in the previous work. there are other works @xcite ; @xcite -that described keystroke savings evaluation. what is more important, the authors worked with agglutinative languages, bahasa (iso-639, ind), the official language of indonesia, and korean (iso-639, kor), official and national language of both north korea and south korea (originally korea). though we do not want to use the same language modelling technics as were described in the papers, we still find it inspiring there are works dedicated to this task. as we mentioned before, we assume that the usage of morphs while doing text prediction will make it both effective and ergonomic; in the same time, morphological segmentation brings new challenges. @xcite , devoted to kunwinjku, a polysynthetic language of northern australia, and turkish, showed that morph-based autocomplete for polysynthetic languages can be troublesome due to long words and sparse vocabularies of such languages. moreover, dialectal variations and dealing with input errors using edit distance makes the next-morpheme predictioning even harder, so, as it is shown in the paper, turkish may be a more attractive language for morph-based predictioning than kunwinjku. ## tasks as mentioned previously, our experiments are split into four distinct tasks, from the more fundamental to the more application-specific. in the following sections we describe the methodology for these tasks and the results obtained. ## segmentation the idea to compare statistical and morphological segmentation was already tested by other researchers; for example, @xcite showed that the usage of morphological segmentation significantly improves the bleu and chrf3 metrics in neural machine translation (nmt). in this paper we want to compare statistical segmentation, presented by unigram @xcite and wordpiece @xcite , and morphological segmentation foot_4 . we choose neu-ralmorphemesegmentation (nms; @xcite for morphological segmentation ## predictive text input in order to evaluate the models we do predictive text input. the idea is that we automatically emulate a person using a smart keyboard while it is offering some predictions, which have to be meaningful. the meaningfulness is important because we assume that the typing person would like to choose from real words/morphs and not some artificial subwords that make at best no sense and in a worst case scenario they may mean something totally wrong (3). the example is given in turkish because it illustrates the problem well. while evaluating, we look through top 3 model predictions and compare them to the subword we are currently predicting. if they are equal, that prediction is chosen, otherwise we look at the next one. if none of the predictions were correct, we consider that the user will have to finish the word character by character. thus, a total number of clicks for a word is computed to measure clicks per character metric: @xmath0, where cpc is clicks per character, keys prediction is the count of predicted clicks (spaces are included), keys normal is the count of clicks needed to input the word character by character. we also include the keystroke savings rate used in @xcite so that we can compare our results with theirs: ## discussion as we can see, the evaluation shows that there is no single model that outperforms the others in both languages, but models that use morphologically segmented data generally show higher scores. thus we recommend to try morphological segmentation as it can be used with a statistical one. it is important to mention is that there is no need in training models using morphologically segmented data from scratch, the existing models can be finetuned and the results will not differ significantly from the ones of nms. k'iche' models in all the tasks have better performance than chukchi models. while we do not know the particular reason for this, we assume that the polysynthetic language complexity may be hindering the model from training. in the mentioned above @xcite the authors also reported that polysynthetic languages have their special challenges such as high word length, complexity, etc. as we reference @xcite , it seems reasonable to compare the results of their experiments with the results of ours. as our task was to predict linguistic units, not any kind of units, while in the mi'kmaq paper words and bpe segments were being predicted, comparison of the results may seem not really correct; though if we do compare the results, we can see that the best ksr score for mi'kmaq is 3.81, while the best score for k'iche' is 4.26. at the same time, the best chukchi ksr (0.79) is much worse that the mi'kmaq score. alongside the metrics we compute there is also a metric which requires end-user testing -the sanity check. as mentioned before, the issue with statistical segmentation is that subwords predicted and offered to the user may have no sense for the user or, what is much worse, may carry the wrong meaning. we do suppose that this alone can be a reason to choose morphological segmentation over the regular one because segmentation task is not done just for itself -it serves a purpose in a larger scheme of things. we think that in case the language model will be used in predictive text setting, where the user experience and user reaction is highly relevant, morphological segmentation should be chosen as a subword tokenisation method, while statistical segmentation may be chosen for machine translation, for example. ## future work we plan to test several other language models and language modelling metrics in order to find out what correlates best with text prediction scores. we find it reasonable to experiment on other languages, for example, nahuatl and yupik, in order to get a better understanding when the use of morphological segmentation is reasonable. another task to do is to run an end-user evaluation of multiple segmentations and determine which units are preferred. in order to do this, we also need to solve the problem of predictive text evaluation that the user has to input the first word character by character -to do this, we will possibly have to combine word level and character level based models.