Bootstrapping Techniques for Polysynthetic Morphological Analysis (2020)

Abstract: Polysynthetic languages have exceptionally large and sparse vocabularies, thanks to the number of morpheme slots and combinations in a word. This complexity, together with a general scarcity of written data, poses a challenge to the development of natural language technologies. To address this challenge, we offer linguistically-informed approaches for bootstrapping a neural morphological analyzer, and demonstrate its application to Kunwinjku, a polysynthetic Australian language. We generate data from a finite state transducer to train an encoder-decoder model. We improve the model by “hallucinating” missing linguistic structure into the training data, and by resampling from a Zipf distribution to simulate a more natural distribution of morphemes. The best model accounts for all instances of reduplication in the test set and achieves an accuracy of 94.7% overall, a 10 percentage point improvement over the FST baseline. This process demonstrates the feasibility of bootstrapping a neural morph analyzer from minimal resources.

## introduction polysynthesis represents the high point of morphological complexity. for example, in kunwinjku, a language of northern australia (iso gup), the word ngarriwokyibidbidbuni contains six morphs: fusion combines multiple grammatical functions into a single morph, leading to large morph classes, and challenging the item-and-arrangement leanings of finite state morphology. incorporation presents a modelling challenge because rule-based methods are unable to enumerate an open class, and machine learning methods need to learn how to recognize the boundary between contiguous large or open morph classes. reduplication is also a challenge because it copies and prepends a portion of the verb root to itself, requiring a nonlinear or multi-step process. tackling these phenomena using finite state transducers (fsts) involves a combination of technical devices whose details depend on subtleties of the morphological analysis (cf. @xcite . there remains a need for more investigation of polysynthetic languages to deepen our understanding of the interplay between the options on the computational side, and the most parsimonious treatment on the linguistic side. morphological complexity leads to data sparsity, as the combinatorial possibilities multiply with each morpheme slot: most morphologically complex words will be rare. furthermore, many morphologically complex languages are also endangered, making it difficult to collect large corpora. thus, polysynthetic languages challenge existing ways of building tools and applications for the communities that speak these languages. in this work we investigate kunwinjku, spoken by about 2,000 people in west arnhem in the far north of australia. members of the community have expressed interest in using technology to support language learning and literacy development. thus, we face the challenge of developing useful language technologies on top of robust models, with few resources and in a short space of time. we envisage morphologically-aware technologies including dictionary interfaces, spell checkers, text autocompletion, and tools for language learning (cf. @xcite . this paper is organized as follows. we begin by reviewing previous work in finite state morphology, low resource morph analysis, neural approaches to morph analysis, and data augmentation for morphological reinflection (sec. 2). next, we describe our existing finite state model for kunwinjku verbs (sec. 3). in section 4 we present a neural approach which addresses gaps in the previous model, including the ability to analyze reduplication and to exploit distributional information. next we discuss our evaluation metrics and our handling of syncretism and ambiguity (sec. 5). finally, the results are presented in section 6, including a discussion of how well the neural models address the shortcomings of the fst model. our contributions include: (a) a robust morphological analyzer for verbs in a polysynthetic language; (b) a method for augmenting the training data with complex, missing structure; and (c) a technique for scoring the likelihood of generated training examples. recently, neural models have gained popularity for morphological processing because they address some of the weakness of fsts: subword modeling shows an ability to remain robust in the face of out-of-vocabulary items, and recurrent neural architectures with attention have shown a capacity to learn representations of context which allow the model to incorporate the notion of long-distance dependencies @xcite . neural morphological analyzers can be developed from training data generated by an fst. these analyzers are more robust, handling variation, out-of-vocabulary morphs, and unseen tag combinations @xcite @xcite . they provide 100% coverage, always providing a "best guess" analysis for any surface form. of course, fsts can be modified to accommodate exceptions and oov morphs, but this requires explicit modelling and usually does not achieve the robustness of neural analyzers @xcite . @xcite found that they could augment their training set by hallucinating new stems, increasing accuracy on their test set by 10 percent. this method involved substituting random characters from the target language's alphabet into the region identified by alignment as the probable root. for the sake of cross-lingual generalizability, their method does not consider language-specific structure. the task of morphological analysis, mapping an inflected form to its root and grammatical specifications, is similar to the task of machine transliteration, mapping a sequence of words or characters from source to target language without reordering. for example in kunwinjku, consider the segmentation and gloss of the verb karridjalbebbehni: since the process of segmenting and glossing the verb does not contain any reorderings, the mapping of surface to glossed forms can be viewed as transliteration. ## a finite state model of kunwinjku finite state transducers have long been viewed as an ideal framework to model morphology @xcite . they are still a popular choice for low-resource polysynthetic languages (cf. @xcite . here we summarize some features of kunwinjku and describe the finite state implementation. ## methods in this section we discuss the approach which leverages an incomplete fst to produce a more robust neural morphological analyzer for kunwinjku. those steps include generating training pairs from an fst, tokenizing the data, resampling from the dataset to simulate distributional signal, hallucinating missing structures into the dataset, and training a neural encoder-decoder model on the resampled data. ## evaluation of the neural models we begin by reporting the performance of the neural models in terms of coverage, accuracy, and precision, so that they can be compared with the evaluation of the fst model, described in section 3.2. additionally, we measure the performance of the neural models in terms of precision (p), recall (r), and f1 on the morph level: for each morph tag in the gold target test set, we calculate p, r, and f1, and then calculate the macro-average p, r, and f1 across all tags in the test set (fig. 9 ). this method is more granular than all-or-nothing accuracy over the entire translated sequence, and allows us to get a better picture of how the models are doing on the basis of individual tags. we observed an issue with syncretic ambiguity which complicates the evaluation process (also noted by @xcite . for example, the pronominal prefix kabindican be glossed: here, the pronominal expresses both the subject and object, and is not explicit whether that subject or object is the 3rd person dual or plural, in any of four possible combinations. the disambiguation cannot be resolved at the level of the isolated verb. our initial experiment with the base data set achieved 100% coverage and 68.3% accuracy on the test set. when confronted by the same problem, @xcite decided to collapse ambiguous tags into an underspecified meta-tag. for example, for the kunwinjku data, we might collapse the four tags above into [3pl.3pl.nonpast]. however, doing so results in a potential loss of information. given the wider sentential context, the pronominal could be possibly be disambiguated, so long as the distinction is preserved and all equally-valid analyses are returned. further, as @xcite point out, in the yupik language it is possible for this ambiguity to exist across other categories which are not easily collapsed. in kunwinjku, an example of this would be the pronominals [1sg.2.past] and [3sg.past] which differ in terms of number and valency, and yet share the same null surface form. their differences are such that they can not be easily collapsed into a single meta-tag. therefore we do not penalize the model for producing any variation of equally valid analyses given the surface form, and for each model we adjust the evaluation for syncretism in a post-processing step. ## results and discussion all of the neural models outperform the fst in terms of accuracy and coverage (fig. 8 ). however, the fst is more precise, and this may be useful in certain application contexts. the best model is base+halluc+resample, which improves on the fst by 10.3 percentage points. on the morphlevel, we see that the neural models containing the hallucinated reduplication data outperform the base neural model (fig. 9 ). we posited that the difficulties encountered by the fst model-namely reduplication, out-ofvocabulary items, and spelling variation-could be at least partially addressed by training a neural model on character and tag sequences, and hallucinating instances of reduplication into the training set. for the most part, this held true, as we see gains across all error classes (cf. sec. 3.2). here we report performance with respect to the three largest error classes: reduplication, oov verbs, and oov nouns. ## conclusion we have shown that complex features of polysynthetic morphology, such as reduplication and distributional morphotactic information, can be simulated in the dataset and used to train a robust neural morphological analyzer for a polysynthetic language. in particular, we showed that a robust neural model can be bootstrapped in a relatively short space of time from an incomplete fst. this work represents a successful first iteration of a process whereby the morphological model can be continually improved. indeed, the concept of bootstrapping a model implies an iterative development story where much of the scaffolding used in early efforts will eventually fall away. for example, once the bootstrapped model has been used to tag verbs containing reduplication, we can confirm the model's high-confidence predictions and retrain. in this second iteration, we may find that we no longer need to hallucinate reduplication because it is sufficiently represented in the new training set. similarly, once we have applied the complete neural model to a corpus of natural text, we will no longer need to approximate distributional information. for researchers developing robust morphological analyzers for low resource, morphologically complex languages, this work represents a template of model development which is well-suited for the context. producing a viable morphological analyzer is the first step towards building improved dictionary search interfaces, spell-checking tools, and computer-assisted language learning applications for communities who speak low-resource languages. the pattern of training robust systems on data that has been augmented by the knowledge captured in symbolic systems could be applied to areas outside of morphological analysis, and is a promising avenue of future exploration.