Minimal Supervision for Morphological Inflection (2021)

Abstract: Neural models for the various flavours of morphological reinflection tasks have proven to be extremely accurate given ample labeled data, yet labeled data may be slow and costly to obtain. In this work we aim to overcome this annotation bottleneck by bootstrapping labeled data from a seed as small as five labeled inflection tables, accompanied by a large bulk of unlabeled text. Our bootstrapping method exploits the orthographic and semantic regularities in morphological systems in a two-phased setup, where word tagging based on analogies is followed by word pairing based on distances . Our experiments with the Paradigm Cell Filling Problem over eight typologically different languages show that in languages with relatively simple morphology, orthographic regularities on their own allow inflection models to achieve respectable accuracy. Combined orthographic and semantic regularities alleviate difficulties with particularly complex morpho-phonological systems. We further show that our bootstrapping methods substantially outperform hallucination-based methods commonly used for overcoming the annotation bottleneck in morphological reinflection tasks.

## introduction the introduction of neural models into natural language processing in the last decade has led to huge improvements in all supervised generation tasks, including morphological inflection. 1 in particular, previous works @xcite have achieved near-perfect performance over the paradigm cell filling problem (pcfp) @xcite , wherein models are required to provide any form in an inflection 1 in recent years the term reinflection has surfaced as a reference to morphological inflection done not necessarily from the lemma. in this paper we will refer to both inflection and reinflection as "inflection", and specify whenever we refer to inflection done exclusively from the lemma. the problem with the first kind of attempts is that given the neural nature of the most successful models, their performance on limited supervision is capped, and data augmentation is likely to help only if the initial data is diverse enough. as for the second scenario of no supervision at all, it is somewhat pessimistic and unrealistic. even if much labeled data for a language does not exist for a low-resourced language, typically there exists knowledge about its paradigm structure that can be employed. unimorph @xcite , for example, includes small amounts of labeled inflection tables for many languages, from obscure ones like ingrian to national languages with widespread usage that lack global attention like georgian. in this work we propose a new, low-resourced morphological inflection scenario, which is more optimistic and realistic for those widely-spoken sparsely-annotated languages. we assume a minimal supervision set and a large bulk of unlabeled text, thus balancing both trends of lowering supervision resources. we bootstrap a tiny amount of as little as five inflection tables, that could be eas-ily written by hand, into a full-blown training set, albeit noisy, for an inflection model trained in a supervised manner. our approach makes use of the regularities abundant in inflectional morphology, both orthographic regularities and semantic ones. based on this method we train morphological inflection models for eight languages and show that, for the five indo-european of them, orthographic regularity is enough to train a morphological inflector that achieves reasonable success. we further show that for languages with complicated morphophonological systems, such as finnish, turkish and hungarian, a method combining both orthographic and semantic regularities is needed in order to reach the same level of performance. an error analysis reveals that the closer an inflection is to the verge of disappearance, the poorer our system performs on it, as less examples exist in the data-derived vocabulary. our models outperform makarov and clematide (2018)'s model designed for low-resourced setting, even when equipped with additional hallucinated data @xcite . we also outperform the best model of @xcite , that didn't use any inflection tables, and their skyline for most languages. we conclude that bootstrapping datasets for inflectional morphology in low-resourced languages, is a viable strategy to be adopted and explored. 3 2 the minimally-supervised setup problem statement let l be a set of lexemes, each corresponding to an inflection table w @xmath0...w l tm } where w l t i is a form of lexeme l bearing the feature bundle t i . our goal is to train an inflection model that maps words bearing one set of features to words bearing another set within the same lexeme. for example, in the french verbal paradigm: indprs1sg, finis , @xmath1, we propose a minimally-supervised scenario where we are only given a small set of n examples of complete inflectional tables @xmath2...w ln } (each of which is of size m), and a large bulk of naturally occurring (that is, in-context) unlabeled data in that language, that is, w 1 .....w c , such that @xmath3. ## the algorithmic framework this work suggests utilizing the patterns exhibited in the small supervision seed, and finding words that exhibit similar (or, analogous) patterns. the algorithm proposed here works in two phases. first we tag words with morphological features if they are found similar (analogous) to examples in the minimal supervision. then, we pair the tagged words such that each pair will include two forms of the same lexeme. this algorithmic division of labor allows the pairing module to provide a sanity check, and reduce the noise potentially lingering from the word-tagging module. the above sketch of the algorithm is quite generic, that is, we can get different instantiations of this framework by plugging in different ways to calculate similarities or analogies between words. in our various algorithmic implementations, we will use the regularities that prevail in inflectional morphology and are detectable even from a tiny amount of supervision. these regularities are manifested both orthographically, as edits between forms tend to repeat themselves across lexemes, and semantically, as forms that share an inflectional function tend to be used similarly. in the rest of this section we will describe both modules, each with its different variants depending on the different notion of similarity used. ## related work in recent years, neural sequence-to-sequence models have taken the lead in all forms of morphological tagging and morphological inflection tasks. morphological tagging is an analysis task where the input is a complete sentence, i.e., a sequence of word forms, and the model aims to assign each word form in-context a morphological signature that consists of its lemma, part-of-speech (pos) tag, and a set of inflectional features @xcite @xcite . morphological inflection works in the opposite direction, and may be viewed as a generation task. here, forms of a lexeme are generated from one another given sets of inflectional features of both the input and output. in many implementations the input form is the lemma, in which case the inflectional features of the input are not given @xcite , and the lemma can be either spelled-out, or inputted as an index in a dictionary @xcite . 13 while most pioneering models for supervised morphological inflection used statistical models based on finite-state-machines (kaplan and @xcite , nowadays neural models for morphological inflections are a lot more pervasive @xcite @xcite ) (and they go as back as @xcite . in the case of unsupervised learning of morphology, a key task is to induce complete paradigms from unlabled texts. early works on unsupervised morphology induction focused on morpheme segmentation for concatenative morphology @xcite @xcite . notwithstanding, early unsupervised works that are not limited to concatenative morphology do exist @xcite . more recent studies on unsupervised morphology include works on knowledge transfer within a genealogical linguistic family well-to lowresourced languages @xcite @xcite , as well as works aimed at modifying the approaches for the supervised problem, to allow better tackling of low resourced scenarios. these include @xcite that focused on data augmentation, and anastasopoulos and neubig (2019) that modified the model itself with a separate features encoder and introduced the now commonly-used hallucination method for data augmentation. the state of the art model for classic low-resourced scenarios, with a diverse but small dataset, is the transition-based model of @xcite . in addition, some works deal with other low-resourced scenarios and assume no inflection tables at all, and are focused on paradigm detection/completion in addition to inflection @xcite @xcite . in contract, our scenario provides the knowledge on the paradigmatic structure with a small and undiverse supervision set. another work that made use of semantics is that of @xcite , who employed analogies between embedded words to filter candidates affixation rules. we use embeddings to discover examples in a more general morphological scenario. ## conclusions in this work we propose a realistic minimallysupervised scenario for morphological inflection, which includes only a handful of labeled inflection tables as well as a large bulk of unlabeled text. we showed that semantic and orthographic regularities allow bootstrapping the minimal supervision set to a large (noisy) labeled data set, by searching for word pairs in the vocabulary analogous to observed form pairs from the supervision. we demonstrate that training a neural morphological inflector over the bootstrapped dataset leads to some non-trivial successes, especially on paradigms of smaller size and on commonly-used inflections. this contribution is orthogonal and can be applied in tandem with hallucination approaches. when applied separately, our method outperforms both hallucination and current state of the art models for lowresourced settings. in the future we aim to improve performance over larger paradigms and rarer forms in order to make our method a viable substitute for the labor-intensive manual annotation for new languages.