Morphological disambiguation from stemming data (2020)

Abstract: Morphological analysis and disambiguation is an important task and a crucial preprocessing step in natural language processing of morphologically rich languages. Kinyarwanda, a morphologically rich language, currently lacks tools for automated morphological analysis. While linguistically curated finite state tools can be easily developed for morphological analysis, the morphological richness of the language allows many ambiguous analyses to be produced, requiring effective disambiguation. In this paper, we propose learning to morphologically disambiguate Kinyarwanda verbal forms from a new stemming dataset collected through crowd-sourcing. Using feature engineering and a feed-forward neural network based classifier, we achieve about 89% non-contextualized disambiguation accuracy. Our experiments reveal that inflectional properties of stems and morpheme association rules are the most discriminative features for disambiguation.

## introduction for morphologically rich languages, morphological analysis and disambiguation plays a critical role in most natural language processing (nlp) tasks. when inflections are generated by piecing together multiple morphemes, a large and sparse vocabulary is produced, requiring tools to unpack the individual morphemes for downstream nlp tasks such information extraction and machine translation. a key characteristic of these languages is that morphemes often have specific meanings (often relating to properties of the words they form or referring to contextual entities) and their combination into words is mostly regular. figure 1 shows typical morphological units contained in the word 'ntuzamwibeshyeho' (never underestimate him/her). while several morphologically rich languages such as turkish, arabic and modern hebrew already have mature tools for morphological segmentation @xcite @xcite , kinyarwanda still lacks appropriate tools for the task. a key limitation in the effort is the need to have high quality datasets manually annotated by language experts. with limited funding opportunities, research on nlp for low resource languages lags behind recent advancements made for nlp on high resource languages. in this work, we leverage an easy to collect stemming dataset and transform it into a resource for morphological disambiguation. while the focus here is on kinyarwanda verbal forms, the method can be applied to other morphologically rich languages. collecting stemming data is much faster and less prone to errors than full morphological segmentations which require subtle linguistic knowledge. through a maximum entropy model, we are able to combine morphological properties of stems with inflectional similarity information from word embeddings to accurately disambiguate candidate segmentations from a rule-based morphological analyzer. our work here pertains to non-contextual verb-phrase disambiguation but is a key step towards contextual disambiguation. we believe that this work will allow rich morphology to be incorporated in new models for kinyarwanda and improve downstream nlp tasks on the language. figure 1: morphological segmentation of the word 'ntuzamwibeshyeho'. each morpheme unit has a specific meaning or function in order to get the meaning of the whole word. the word is an inflection of the lemma 'kwibeshya' (to err) which a derivation from 'kubeshya' (to lie) by adding a reflexive -ii-. therefore, a literal morpheme-by-morpheme translation of the word would be 'never lie to yourself about him/her' while the real meaning is 'never underestimate him/her'. ## related work computational morphology has been studied for decades, but most implementations and evaluations have been conducted on languages that are not related to kinyarwanda. finite state methods for morphological analysis have been proposed by beesley and karttunen @xcite and have been popular for morphological analysis. our morphological analyzer is based on the underlying principle of two level morphology @xcite , but our custom implementation does not follow the exact formalism of finite state transducers. we rather focus on refining rules that are specific to kinyarwanda through extensive empirical examination. in @xcite , a morphological alternation model for kinyarwanda was presented using xerox tools, but no empirical evaluation was conducted. in @xcite , an experiment was conducted on learning pos taggers for kinyarwanda and malgasay using a small dataset of frequent words annotated by linguists. while pos tagging is an important task, morphological analysis is even more important for morphologically rich languages because it reveals more information than what can be covered by a finite set of tags. other work on kinyarwanda has been more linguistic in nature @xcite ; @xcite , especially due to that kinyarwanda is considered as a more generic prototype of the larger group of bantu languages, owing to its rich morphology and tonal system. the problem of morphological disambiguation has been researched on for other morphologically rich languages such as turkish, arabic and modern hebrew. proposed methods range from statistical ones @xcite , to rule based approaches @xcite , and more recently using recurrent neural networks @xcite . most of these methods are usually trained and evaluated on richly annotated tree-banks which are not available for low resource languages like kinyarwanda. the only labeled data required by our method is a list of inflection/stem pairs which can be conveniently collected by untrained native speakers. another major difference is also that our disambiguation focuses on uncontextualized morphology of kinyarwanda verbal forms. we reserve contextual disambiguation and part of speech tagging for a future study. figure 2 : a smart-phone interface is used by volunteers to stem kinyarwanda verb forms. for instance, the analysis of the inflected form 'izarigishijwe'(...which have been made disappear) can result in ambiguous stems -'igish'(kwigisha: to teach), 'gish'(kugisha: to ask [for advice]), 'rigis'(kurigisa: to make disappear), 'ig'(kwiga: to study), 'shir'(gushira: to end) and 'siz'(gusiza: to prepare a place for construction) 3 methods ## --ka -----tu -ik --ir ---w a - step 3. generate inflection sets s @xmath0: the morphological analyser formulates a set of common inflections for each candidate stem: ## experimental setup the first step in our experiments was to generate stem morphological indicator features from user annotations as explained in section 3.4. the second step involves preparing the dataset for training and evaluation. after features are extracted, we split the data between in training and validation set and then train a baseline classifier using only the data from user annotations. we up-sample by 4 factors the annotations from the best trained annotator who is equipped with subtle linguistic understanding of kinyarwanda verb morphology. we use the baseline classifier to then predict the stem for the entire unlabeled vocabulary of kinyarwanda verbal forms. we rank the predicted stems by prediction uncertainty (entropy). the most uncertain instances are sent back to annotators for labeling in a batched active learning fashion @xcite . for active learning, we send batches of the top 10000 uncertain samples for which the entropy @xmath1. we also enrich our labeled set with the most confident predictions in a semisupervised manner. for semi-supervised learning, we only take examples for which the baseline model has labeled with at least 0.95 top probability (p 1 ), having at least 3 competing stems, (p 1 -p 2 ) > 0.95 and entropy @xmath2.1. after expanding our labeled data through active learning and semi-supervised learning, we repeat the first step of feature extraction to form our final training and evaluation dataset, which contains about 170,000 examples. we split our dataset into training, development and test set in the ratio of 70%+15%+15% respectively. all our models are trained with gradient descent using adam update rule (kingma and ba, 2014) using 0.01 learning rate in 256-sized mini-batches and for 50 epochs. for our best fine-tuned model, we re-train the model with large batches of 4000 examples each using lamb method @xcite for 100 more epochs. since all features are precomputed, training each feedforward neural network takes less than 5 minutes on an quad-core machine. we use posix c and c++11 for this project with the only external dependency being eigen matrix algebra library foot_0 . ## experimental results and discussion model size -we evaluated the model robustness by varying the number of hidden units in the feedforward neural nets ( ## conclusion this work focused on morphological disambiguation of kinyarwanda verb forms using maximum entropy model on new crowd-sourced stemming dataset. high disambiguation accuracy was achieved through careful feature engineering. intuitively curated inflectional features emerged as important parsimonious predictors. future work should look at how to directly use morpheme embedding methods as a way to more generically represent both semantics and morphology in a unified form. achieving total disambiguation ultimately requires complete contextual information which may not be available in written form.