Open-Source Morphology for Endangered Mordvinic Languages (2020)

Abstract: This document describes shared development of finite-state description of two closely related but endangered minority languages, Erzya and Moksha. It touches upon morpholexical unity and diversity of the two languages and how this provides a motivation for shared open-source FST development. We describe how we have designed the transducers so that they can benefit from existing open-source infrastructures and are as reusable as possible.

## introduction there are over 5000 languages spoken world wide, and a vast majority of them are endangered (see @xcite . the mordvinic languages erzya and moksha are no exception. one of the first nlp solutions that are typically developed along with lexical resources for any low-resourced language is a morphological analyzer (cf @xcite @xcite . in this paper, we describe the development of an open-source fst (finite-state transducer) based morphological analyzer, lemmatizer and generator for erzya and moksha. we highlight the importance of certain design decisions to ensure the compatibility of our transducers with existing systems. in addition, we will describe how the transducers can be edited in a system that creates an abstraction layer behind a graphical user interface for fst development. the unity and diversity of the mordvin literary languages of today, erzya and moksha, has been a subject of research for over two hundred years. the first grammars were published in 1830s -moksha in 1838 @xcite and erzya in 1838-1839 @xcite . the subsequent 180 years brought scholars for fieldwork, grammars, dictionaries, and the popularization of the languages. 2002 saw the publication of the first monolingual dictionary of erzya @xcite , and the manuscript was proclaimed open by the author for future development. the mordvin languages have continued to receive a fair share of linguistic research interest in the recent years @xcite @xcite . after the release first finite-state transducer for the closely related komi-zyrian @xcite , it was only obvious that similar work should be done for erzya mordvin. fortunately, over the past decade there has been an increasing number of publications on erzya, relating to its morphology @xcite , its ocr tools @xcite and universal dependencies @xcite . this document discusses open-source morphology development, which has greatly benefited from open-source projects, most notably achievements attributed to the giellalt infrastructure @xcite , i.e. giellatekno & divvun at the norwegian arctic university in tromsø, norway. it is also very important that people new to language documentation be given opportunities to develop their understanding of languages through participation in projects. here we must mention an important span of time 1988-1997, during which the first author did word processing for the 2073page 'dictionary of mordvin dialects' by heikki paasonen. the transducers are available on github for erzya foot_0 and moksha foot_1 . the nightly builds are available through a python library called uralicnlp foot_2 @xcite . rule-based morphology can be implemented in many different systems. popular tools include xfst @xcite , foma @xcite and openfst @xcite . however, we use hfst (helsinki-finite state technology) @xcite because it is the system used in @xcite . giellalt provides our transducers with a list of quality attributes. their infrastructure consists of work done for multiple endangered languages in such a way that the morphological resources can be used in a multitude of different contexts such as disambiguation @xcite , dependency parsing @xcite , online dictionaries @xcite , spell checkers @xcite , online creative writing tools @xcite , automated news generation @xcite and language learning tools @xcite . in order to gain the added benefit from the giel-lalt infrastructure, we have to design our transducers so that they are compatible with hfst and that they follow a certain morphological tagset and that they take the input and output in a certain format. these requirements define the api our transducers need to implement. an example of this can be seen in apertium @xcite is another opensource system that uses fst transducers for rulebased machine translation. they use their own transducer format, but fortunately also support hfst based transducers, and in fact apertium type transducers can be compiled to hfst form as well @xcite . however, their tagset is different from that of giellalt. this is solved by applying filters at the time the fst is compiled to produce a separate, apertium-compatible, transducer automatically. ## erzya and moksha language pair at the moment erzya and moksha infrastructure in giellalt can be considered to be in nearly equal standing, despite the fact that work on erzya was started considerably earlier. this does not mean, however, that the resources for both languages are identical in all measures, although in basic numeric levels their sizes are comparable. there are many situations where additional consistency between the infrastructures for these two languages would be quite desirable and beneficial. we discuss next what these instances could be, and outline some of the major questions while working with this language pair. the open-source machine translation infrastructure apertium uses a shallow-transfer strategy. by definition, shallow transfer makes a morphosyntactic analysis of the source language, translates the lemmas from source to target language, generates morphology acquired from the source language in the target language and makes adjustments to the syntax of the target language. this concept of parallel lexica, morphology and syntax, would therefore, seem most effective in translation between closely related languages. in the case of the mordvin pair, this means the use of mutual tags for describing mutual phenomena. mutual phenomena in erzya and moksha, however, can be distinguished as exact matches and fuzzy matches, as it were. while there is no doubt that the two languages share nearly the same categories of case, person, number and definiteness, it must also be noted that the paradigms do not share the same cellular structure @xcite @xcite . when the paradigm structure is diverse, it is suggested that a union of morphological tags be taken as a starting point. when one language distinguishes a category of and, in order to facilitate a transition from the absence of the category of number to its presence and vice versa, the rudiments of tagging this category must be put in place, e.g. for the category of number we use sg 'singular', pl 'plural' and sp 'singular or plural'. diversity in the erzya-moksha language pair can be found in the core ranges of the categories definiteness, person and number. both languages have an indefinite or basic declension, a definite or determiner declension and a possessive declension. for both languages, it can be stated that the indefinite declension distinguishes the category of number in the nominative alone, whereas number is always specified in the definite declension paradigms. the possessive declension, however, exhibits a salient rift in unity. moksha has a symmetric paradigm in the core cases, nominative, genitive and dative (see additional diversity is found in the subjectobject paradigm, where portmanteau morphology enables the specification of first, second and third person subject and object provided both arguments are singular. when one or both of the arguments is not specifically singular, however, one form may serve to indicate more than one set of arguments, e.g. erzya has a default non-past form -samiź which simply indicates a first person object with a second or third person subject when it is not true that both subject and object are specified, singular entities the differences outlined here are not the same, but comparable, to those found in a recent study that evaluated the morphological differences between komi-zyrian and komi-permyak within the context of fst and treebank development @xcite . with closely related languages any kind of resource reuse or transfer is rarely trivial, but through careful evaluation of linguistic features and differences we show that these goals are certainly possible and realistic. ## current state and ensuring maintainability at the time of writing the transducers lemmas, stems and glosses were acquired through several channels. statistics on the coverage of the erzya fst can be seen on moksha are shown in although the largest bilingual dictionaries for erzya and moksha provide declension information, classification of declension for other nominals has not been dealt with. in the noun phrase, adjectives are declined only when they are promoted to np head in ellipsis or in the predicative. hence, adjectives, by nature, might be declined to virtually the same extent as common nouns, but they are subject to fewer derivations. proper nouns, although most commonly encountered in the singular, may, in fact, be declined in the plural -place names when declined in the plural are down cased and their new semantics refer to inhabitants. person names gain a sense of associative plural. derivations include morphologically new forms and orthographic compounds. most salient are the ever present diminutive, which has a sense of endearment, diminutive and even partitive measure. orthographic derivation can be observed in compound nouns, with collective sense, and verbs, expressing simultaneous activities. the challenge of orthographic compounds lies in the hyphened pair where both elements are inflected for the same grammatical categories, such that copula forms of compound nouns are included in the morphology of both stems, but the additive clitic attaches only to the second. the transducer development is conducted in an agile fashion with nightly builds available for the end user via an open-source python library called uralicnlp @xcite . the quality of the transducers is ensured by unit testing. we use test scripts written in yaml that contain a list of inputs and accepted outputs. any changes in the transducers that break the tests will be immediately noticed and acted upon. the source code of an fst is not the easiest to write for an average linguist or an nlp researcher. in the context of endangered languages, however, one would hope that people working on language documentation with a limited technical background or even community members could make an active contribution to the morphological tools. in order to address this need, we have embraced two levels of abstraction for fst development. the first level of abstraction is the use of an xml-based dictionary. lexical data is stored in xml form in the giellalt infrastructure, and to a great extent, they contain similar information to the lexicon of an fst: words (lemmas and stems) and their continuation lexicons that indicate how they are inflected. for this reason, we actually compile the fst lexicon from the xml-based dictionary. the xml dictionaries are, in fact, also used as source files for online dictionaries, and therefore they are an attempt to address the matter of reuse, i.e. transducer construction, online morphological dictionaries, source material for icall projects as well as rule-based machine translation. with a recent effort of a tei (text-encoding initiative) compatibility layer in the giellalt xml dictionaries @xcite , it was decided that editing the fsts in the xml dictionaries should be open for a majority of people using modern lexicographic tools. this compatibility layer is important as tei is an iso-standard, which means that it will most likely outlive any individual project and remain usable in the future as well. the second layer of abstraction is the opensource online user interfaces ve'rdd @xcite and akusanat @xcite that make it possible to edit xml based dictionaries for anyone without any technical background. this is important since direct edits to an xml might be daunting for a language community member who has no programming background. these two different levels of abstraction ultimately produce xmls in the giellalt format that can be directly used to enhance the transducers. to provide further comparison, a very similar mechanism to store lexicon externally from the fst has also been used successfully by @xcite with pite saami. also in this case same lexicon is used in fst, in a published dictionary @xcite , and in an online dictionary 4 . this shows that approach described here is practical, and already used, in different endangered language contexts. ## conclusions in this paper we have presented our work on opensource fsts for erzya and moksha. leveraging existing open source platforms has been the core design principle throughout the development. without giellalt support, our transducer would be left out of a plethora of higher level services provided by the infrastructure such as spell-checking, constraint grammar based syntax, language learning tools etc. this would mean that we would need to build these resources from the ground up. another important design principle has been accessibility of the fsts. continuous integration makes it possible to commit to the nightly builds of the uralicnlp library, which makes our fsts usable through a generic multilingual python api. accessibility has also been thought of from the point of view of the development. using an xml based dictionary to generate the fst lexicon with a compatibility with the iso-standardized tei xml makes it easier for non-fst savvy people to contribute to the work. furthermore, integration with a gui such as ve'rdd, makes it possible to crowdsource the development in the future by evoking community involvement of the native speakers. we have had positive experiences from building on top of these other open-source solutions and would strongly recommend that other researchers working on the morphology of endangered languages investigate the existing open-source platforms before starting to build everything from scratch. the giellalt infrastructure could save you from two to three years of development. at the same time attention has to be paid to general apis and interfaces that allow access to these tools at various levels of abstraction.