Neural Morphology Dataset and Models for Multiple Languages, from the Large to the Endangered (2021)

Abstract: We train neural models for morphological analysis, generation and lemmatization for morphologically rich languages. We present a method for automatically extracting substantially large amount of training data from FSTs for 22 languages, out of which 17 are endangered. The neural models follow the same tagset as the FSTs in order to make it possible to use them as fallback systems together with the FSTs. The source code, models and datasets have been released on Zenodo.

## introduction morphology is a powerful tool for languages to form new words out of existing ones through inflection, derivation and compounding. it is also a compact way of packing a whole lot of information into a single word such as in the case of the finnish word hatussanikinko (in my hat as well?). this complexity, however, poses challenges for nlp systems, and in the work concerning endangered languages, morphology is one of the first nlp problems people address. the giellalt infrastructure @xcite has @xcite finitestate transducers (fsts) for several morphologically rich (and mostly uralic) languages. these fsts are capable of lemmatization, morphological analysis and morphological generation of different words. these transducers are at the core of this infrastructure, and they are in use in many higher level nlp tasks, such as rule-based @xcite and neural disambiguation @xcite , dependency parsing @xcite and machine translation @xcite . the transducers are also in constant use in several real world applications such as online dictionaries @xcite , spell checkers @xcite , online creative writing tools @xcite , automated news generation @xcite , language learning tools @xcite and documentation of endangered languages @xcite . as an additional important application we can mention the wide use of fsts in the creation of universal dependencies treebanks for low-resource languages, at least with erzya @xcite , northern saami @xcite karelian @xcite and komi-zyrian @xcite . especially in the context of endangered languages, accuracy is a virtue. rule-based methods not only serve as nlp tools but also as a way of documenting languages in a machinereadable fashion. members of language communities do not benefit, for example, from a neural spell checker that works to a degree in a closed test set, but fails miserably in real world usage. on the contrary, a rule based description of morphology can only go so far. new words appear and disappear all the time in a language, and keeping up with that pace is a never ending job. this is where neural models come in as they can learn to generalize rules for out-of-vocabulary words as well. @xcite also showed recently that at least with finnish the neural models do outperform the rule-based models. this said, finnish is already a larger language, so the experience doesn't necessarily translate into low-resource scenario (see @xcite . the purpose of this paper is to propose neural models for the three different tasks the giel-lalt fsts can handle: morphological analysis (i.e. given a form such as kissan, produce the morphological reading +n+sg+gen), morphological generation (i.e. given a lemma and a morphology, generate the desired form such as kissa+n+sg+gen to kissan) and lemmatization (i.e. given a form, produce the lemma such as kissan to kissa 'a cat'). the goal is not to replace the fsts, but to produce neural fallback models that can be used for words an fst does not cover. this way, the mistakes of the neural models can easily be fixed by fixing the fst, while the overall coverage of the system increases by the fact that a neural model can cover for an fst. the main goal of this paper is not to propose a state of the art solution in neural morphology. the goal is to first build the resources needed to train such neural models so that they will follow the same morphological tags as the giellalt fsts, and secondly train models that can be used together with the fsts. all of the trained models will be made publicly available in a python library that supports the use of the neural models and the fsts simultaneously. the dataset built in this paper and the exact train, validation and test splits used in this paper have been made publicly available for others to use on the permanent archiving platform zenodo. ## constructing the dataset we are well aware of the existence of the popular unimorph dataset @xcite . however, it does not suit our needs of two reasons. one reason is the incompatible morphological tagset. our goal is to build models that can directly be used side-by-side with the existing fsts, which means that the data has to follow the same formalism. conversion is not a possibility, as the main reason we are not interested in using the uni-morph data is its limited scope; not only does it not cover all the languages we are dealing with in this paper, but it does not cover any cases of complex morphology. for example, the finnish dataset does not cover possessive suffixes, question markers, comparative, superlative etc. such a data would not be on par with the output produced by the fsts. we produce the data for the following languages: german (deu), kven (fkv), komi-zyrian (kpv), mokhsa (mdf), mansi (mns), erzya (myv), norwegian bokmål (nob), russian (rus), south sami (sma), lule sami (smj), skolt sami (sms), võro (vro), finnish (fin), komi-permyak (koi), latvian (lav), eastern mari (mhr), western mari (mrj), namonuito (nmt), olonets-karelian (olo), pite sami (sje), northern sami (sme), inari sami (smn) and udmurt (udm). a vast majority of these languages are greatly endangered @xcite . we use the fsts and dictionaries from the giel-lalt with the uralicnlp @xcite library to build the datasets for training the models. we do this in a clever way by taking all open class part-of-speech words from the dictionaries for each language and use the fsts to produce all morphological readings for them. the number of words in the giellalt dictionaries is shown in the reason why we do the testing purely on outof-vocabulary words is simply to test the accuracy of the models in the scenario that is more close to the one they are trained for, namely, in cases where the fsts fail in their coverage. ## experiments and results in this section, we cover the neural architecture for the three separate morphological tasks: lemmatization, analysis and generation. we also show the results of the models in these tasks for each language, and present an error analysis on the finnish and komi-zyrian by taking a closer look at the results. ## conclusions in this paper, we have presented a method for automatically extracting inflectional forms from fst transducers by composing a regular expression transducer for each word with an existing fst transducer. this way, we have been able to gather very large morphological training data for analysis, lemmatization and generation for 22 languages, 17 out of which are endangered and fighting for their survival. we have used this dataset to train neural models for each language. because the data follows the tags and conventions used in the giellalt infrastructure, these neural models can be used directly side by side with the fst transducers in many of the applications that depend on them. the results look very good for some languages while being a bit more modest for others. analysis seems to be the hardest problem out of the three, and its training also took the longest time. despite this, many models reached to an over 80% accuracy in the tasks. this is rather good given that the evaluation was conducted entirely on out-ofvocabulary words. the accuracies reported in this paper are a somewhat lower than what they could be. this is due to the fact that we ran the evaluation by producing one result only for each input with the neural models and compared that input directly to the one in the test data. as we saw in our analysis, many of the inputs in the test data were ambiguous, which caused the neural model to produce an output that is correct, but not the one in the test data. however, the right way to overcome this problem would be to research how to deal with ambiguity. the neural models we trained can already now produce n best candidates for each input. it is probable that within those n best candidates, the models actually cater for the ambiguity and produce other results that are correct as well. for instance, the finnish word noita, can be an accusative singular noun meaning 'witch' or a partitive of nuo meaning 'them'. knowing how to maximize the number of forms the neural model produces while minimizing the number of incorrect forms is a question for another paper. although, some methods could already be used with the models trained in this paper by in-troducing simple modifications to how the results are predicted @xcite . even though we aimed for a real world scale morphological tag complexity by querying all possibilities from the fsts, there are still a couple of morphological categories we did not tackle for practical reasons. one of them is the use of clitics. the problem with these is that they can be attached to almost any kind of word regardless of its partof-speech and inflectional form. on top of this, multiple clitics can be added one after another. to give an idea of the scale, with clitics, finnish has 9425 unique forms for nouns (instead of 850), 216 for adverbs (instead of 16), 14794 for adjectives (instead of 1244) and a whopping 88044 forms for verbs (instead of 6667). this means that clitics need to be solved by taking a different approach than the one we had. one could, for example, introduce some forms with different combinations of clitics here and there in the training data, in which case the question arises on how many forms need to appear with clitics in order for the model to generalize their usage. compounds and derivations could not be included because of how the fsts were implemented. if you ask an fst for compounds and derivations, you will surely get them! even in such quantities that your computer will run out of ram and swap memory for the forms of a single word, as there is no limit to how many words can be written together to form a compound or how many times one can derive a new word from another. we people might have our cognitive limits for that, but the fsts will not 4 . the problem of compounds is probably best to leave for a separate model to solve, as there are already methods out there for predicting word boundaries @xcite . the compound splits by such methods could then be fed into the neural models trained in this paper. as for derivations, some of them could be included in the training data, but the question of how many forms are needed would still require further research.