Unsupervised morphological segmentation in a language with reduplication (2022)

Abstract: We present an extension of the Morfessor Baseline model of unsupervised morphological segmentation (Creutz and Lagus, 2007) that incorporates abstract templates for reduplication, a typologically common but computationally underaddressed process. Through a detailed investigation that applies the model to Maori, the ̄ Indigenous language of Aotearoa New Zealand, we show that incorporating templates improves Morfessor’s ability to identify instances of reduplication, and does so most when there are multiple minimally-overlapping templates. We present an error analysis that reveals important factors to consider when applying the extended model and suggests useful future directions.

## introduction unsupervised models that can learn to segment words into morphemes without requiring extensive hand-written rules have two important advantages (see @xcite . first, their unsupervised nature allows them to capture a key facet of human morphological learning: learning despite the lack of both direct and negative evidence. second, their lack of hand-written rules makes them very flexible: they can be deployed in a range of applications, across diverse languages. however, in order to learn effectively, unsupervised models must make general assumptions about underlying morphological processes, and their success in part reflects the appropriateness of these assumptions for the language(s) under investigation. this can cause the underlying assumptions to become tuned to the morphological processes of high-resource languages used in development and evaluation @xcite , leading models to overlook processes that do not occur in such languages, even if they are typologically common. recent work has highlighted the advantages to such models of incorporating expert linguistic knowledge, such as language-specific morphemes and/or abstract morphological templates @xcite @xcite . we explore the value added to a standard baseline model, morfessor @xcite , by incorporating templates for reduplication, a typologically common but computationally underaddressed process. we conduct a detailed analysis of the successes and challenges in using an enriched model to capture reduplication in māori (polynesian), the indigenous language of aotearoa new zealand, which reveals a promising path for unsupervised morphological segmentation of languages with reduplication more broadly. ## experiments on complex words to see how incorporating reduplication templates affects segmentation of morphologically complex words, we now compare the extended model (all templates) with the original (no templates) on a broader subset of training data, examining their agreement with fluent-speaker segmentations. ## conclusion we have described a method to incorporate abstract reduplication templates into the morfessor baseline model of unsupervised morphological segmentation @xcite . our test on māori shows three main results. first, incorporating templates allows morfessor to better identify instances of reduplication. second, the more distinct templates incorporated, the better the model performs. third, the benefits of incorporating additional templates are strongest for items matching those templates, but also present for items matching other templates, due to the pooling of statistical support for the reduplicant morph, red. we have also discussed factors that should be considered when applying the extended model. first, care should be taken to minimize interactions between templates, to avoid competition that coerces multiple kinds of reduplication to the same template. second, the training set should be as large and as similar to the test set as possible, because coercion between templates is more prevalent in the viterbi algorithm used for untrained items than it is in the recursive algorithm used for trained items. third, the training set should include both reduplicated forms and their (apparent) bases of reduplication, as excluding the base can preclude it from being identified in the reduplicated form, which can in turn increase the risk of coercion to an incorrect reduplication template. our results clearly show the value of incorporating expert linguistic knowledge into unsupervised morphological segmentation. we have shown how this improves segmentation of reduplicated words in māori, while still permitting accuracy on nonreduplicated words. while we have focused on māori, we expect performance gains to transfer to other polynesian languages with similar reduplication templates, and we expect the higher level modeling approach and insights to extend more broadly to any language that has productive reduplication processes. given the high typological prominence of reduplication @xcite , the incorporation of reduplication templates offers a promising avenue for improving the cross-linguistic adequacy of unsupervised morphological segmentation.