Modeling Morphological Typology for Unsupervised Learning of Language Morphology (2020)

Abstract: This paper describes a language-independent model for fully unsupervised morphological analysis that exploits a universal framework leveraging morphological typology. By modeling morphological processes including suffixation, prefixation, infixation, and full and partial reduplication with constrained stem change rules, our system effectively constrains the search space and offers a wide coverage in terms of morphological typology. The system is tested on nine typologically and genetically diverse languages, and shows superior performance over leading systems. We also investigate the effect of an oracle that provides only a handful of bits per language to signal morphological type.

## introduction morphological analysis aims to identify languages' word-internal structures. early approaches to the computational analysis of morphology modeled the structure of each language with hand-built rules, (e.g. @xcite . such systems require a significant amount of work from domain experts, and while they tend to be very accurate, they also suffer from low coverage. supervised and semisupervised machine learning approaches require expert input and will suffer from out-of-vocabulary problems. this paper focuses primarily on fully unsupervised morphological learning, which offers the most flexibility and can be deployed for new languages with no data annotation. concatenation-based morphological learning systems aim to identify morphemes or morpheme boundaries within words @xcite @xcite @xcite @xcite . the morpho-challenge tasks 1 provide a set of morphologically annotated 1 http://morpho.aalto.fi/events/morphochallenge/ data for testing concatenation. however, systems designed directly for identifying morpheme boundaries are limited in that non-linear structures such as infixation cannot be well captured. another approach exploits morphological relations between word pairs. related words form morphological chains through processes of derivation. there are many such processes including affixation at the edges or middle of a word, reduplication, stem transformations, and so on. of these, only edge-affixation is available to concatenation-based models, so leveraging derivation directly allows for wider cross-linguistic coverage @xcite @xcite @xcite . a more holistic line of work builds learning on the concept of morphological paradigms @xcite @xcite . paradigms can be defined as sets of morphological processes applicable to homogeneous groups of words. for example, the paradigm (null, -er, -est, -ly) in english can be applied to adjectives (e.g., high, higher, highest, highly), while (null, -ing, -ed, -s, -er) is defined over verbs (e.g, walk, walking, walked, walks, walker). paradigms have several merits. first, they provide a principled strategy for tackling the data sparsity problem. in morphologically rich languages, a single word can derive hundreds of forms most of which will be unattested in real data. this can be addressed by taking paradigms into account because if a word appears in part of the paradigm, it likely can appear in the rest too. the recent sigmorphon shared tasks in paradigm filling are along this line @xcite @xcite . second, paradigms can be used to identify spurious morphological analyses. for example, the words within, without, wither might be analyzed as applying suffixes -in, -out, -er to the word with, however, the paradigm (-in, -out, -er) is not reliable since it only applies to one single word, i.e. with. one thread common in previous work is the lack of consideration for characteristics of languagespecific morphological typology. in this paper, we propose a new framework that incorporates typological awareness by explicitly modeling different morphological patterns including suffixation, prefixation, infixation, and reduplication. these patterns have covered most common morphological processes of the languages in the world, with the exception of templatic morphology which is not represented in the ldc-provided test sets. by building such universal linguistic knowledge, the model will benefit from both constraining the search space (without generating a large amount of spurious analyses) and providing a wider coverage especially for the non-linear morphological structures. ## related work the morpho-challenge tasks held between 2005 and 2010 motivated a large amount of work on unsupervised morphology learning including the morfessor family of models. the morfessor baseline system @xcite , an mdl model, is one of the most popular unsupervised systems for automatic morphological segmentation. @xcite extend the model with the maximum a posteriori (map) on both observed data and the model. these systems only require word lists as input, which is an advantage for low-resource languages where there is no large corpus for training complex models. various work has explored the idea of paradigms. @xcite try to learn inflectional paradigms on english verbs, @xcite exploits the mdl principle to learn paradigms (referred to as signatures) with a greedy search strategy, and @xcite adopt a semi-supervised log-linear model to identify paradigms, which requires a number of seed paradigms for training. however, in morphologically rich languages such as turkish where a single paradigm can be extremely large, this method requires considerable human annotation effort. @xcite use a semi-supervised approach to learn abstract paradigms from a given inflection table. however, the task is different from what we discuss here, which discovers inflection tables as an intermediate step. @xcite create paradigms from the results of a probabilistic model and use the reliable paradigms to prune unre-liable ones and achieve promising results. @xcite 's model only deals with suffixation. the framework that we develop in this paper is most directly inspired by @xcite . @xcite use semantic information to identify real morphological pairs from a set of orthographically similar word pairs. similarly, @xcite use orthographic information to generate candidate morphological rules, e.g., prefix : $ : in, and then use word embeddings to evaluate the qualities of the rules. @xcite create morphological chains, e.g., (play, playful, playfully), using both orthographic information and distributional semantics by maximizing the likelihood through a loglinear model. one drawback of using distributional information is that it requires large text corpora to train reliable semantic vectors. this is a major hurdle for applying such a system to low-resource languages. based on the output of @xcite 's model, @xcite adopt integer linear programming (ilp) to find globally optimal paradigms, which they call morphological forests, and achieve improved performance. ## morphological typology this section surveys the morphological phenomena frequently observed among the world's languages which our system is able to account for. ## modeling morphological processes in this section, we describe our framework for modeling language morphologies, including prefixation, suffixation, infixation, full and partial reduplication. we also model stem changes that typically occur at word boundaries except for vowel changes. ## disambiguation with a probabilistic model after generating all candidate analyses for a given word, we evaluate how good each candidate is so we can choose the best one as the final analysis. we compute the conditional probability of a candidate analysis [g, m](r) given a word @xmath0, m](r)), namely p (r, g, m|w). p (r, g, m|w) = 0 if [g, m](r) = w. otherwise, we use the following formula to calculate this probability. to compute the probability of a candidate analysis (@xmath1, m](r)), p (r, g, m), we assume that r, g and m are independent to each other. so, the probabilities in this model can be estimated using em initialized by counting all the candidate analyses of all words in the word list and assuming that each candidate has the same probability. ## experiments 6.1 settings we compare our model with morfessor @xcite , the most popular baseline, morpho-chain (mc) @xcite and its improved version, morph-forest (mf) @xcite , and parama (pma) @xcite . we evaluate the models with segmentation points (boundaries of morphemes), the same metric used by @xcite and @xcite . we run our model in two different settings. in the primary experiment, we run it as a fully unsupervised model (fu), assuming all possible typological features. in a secondary experiment, each language's morphological typology is provided by an oracle so that the model can only search relevant patterns per language (u+t). a vowel inventory is also provided so that our system can discover the vowel change rules described in section 4.3. mc and mf are run in two different configurations, one with semantic vectors (+v) and the other without vectors (more comparable to morfessor, parama, and our system). we conduct the experiments with a data set containing 9 languages from diverse language families @xcite . the details of the data sets including the typological features for each language and the size of corpus that is used for training word vectors are shown in ## conclusion and future work in this paper, we develop a model for morphological analysis that exploits typological features to achieve the best performance on a wide range of languages. the tool is publicly available here: https://github.com/xuhongzhi/parama2 . this unsupervised model can be quickly and easily extended to novel languages without data annotation or expert input. combined with the ability to process infixation and reduplication, our system improves access for geographically diverse lowresource languages. although the evaluation is based on segmentation points, our model outputs much richer structure. it can also tell us the productivity of each morphological process and thus can obtain much deeper knowledge in terms of morphological structures of languages. our next step will be to attempt to automate the determination of language typology, yielding somewhat better performance with a system requiring no human intervention per language at all. future work will aim to extend the current model to capture particularly challenging morphological patterns such as templatic non-concatenative morphology and polysynthetic composition.