Representation of Y ine [ A rawak] Morphology by Finite State Transducer Formalism (2021)

Abstract: We represent the complexity of Yine (Arawak) morphology with a finite state transducer (FST) based morphological analyzer. Yine is a low-resource indigenous polysynthetic Peruvian language spoken by approximately 3,000 people and is classified as ‘definitely endangered’ by UNESCO. We review Yine morphology focusing on morphophonology, possessive constructions and verbal predicates. Then we develop FSTs to model these components proposing techniques to solve challenging problems such as complex patterns of incorporating open and closed category arguments. This is a work in progress and we still have more to do in the development and verification of our analyzer. Our analyzer will serve both as a tool to better document the Yine language and as a component of natural language processing (NLP) applications such as spell checking and correction.

## introduction yine is a low resource indigenous polysynthetic peruvian language of the arawak family spoken by approximately 3,000 people living near the ucayali and madre de dios rivers, tributary rivers of the amazon. yine is considered "definitely endangered" according to the unesco atlas of the world's languages in danger @xcite . as noted by @xcite , although yine has a typologically oriented descriptive grammar, documentation and further study of several grammatical aspects are still urgently needed since the yine language is at risk of entering into an obsolescent and consequently disappearing status. therefore, such work is vital to not only adequately document the yine language, but also to support its continued vitality through computer assisted tools such as spell-checkers and machine translators. formal and computational representation of morphology is considered a "solved problem" based * authors contributed equally on beesley and karttunen's work and seminal finite state morphology text @xcite . this does not mean that representing a language is either easy or fast, especially for the case of polysynthetic languages such as yine. our goal is to construct a high coverage finite state transducer (fst) morphological analyzer both to document and preserve the yine language, and to use it in nlp applications, such as spell checking and correction, that might promote language vitality. our contributions at this point are: 1. a partial functioning morphological analyzer for nominal and verbal constructions including possessive constructions and verbal predicates, and 2. various project decisions and fst patterns employed so far in construction of the analyzer. given the incomplete implementation, it is too early to report meaningful project results. representation of yine morphology by a fst is a work in progress. this paper describes relevant morphological features of yine, representation of these features by fst, particularly challenging representation problems, a preliminary evaluation, and our current and planned future states. while we do not apply our work to spell checking in this paper, that is one of our planned goals. previously we had attempted to develop a hun-spell foot_0 based spell corrector, but found it too limiting given the polysynthetic nature of the yine language. this is consistent with @xcite , who found that fst correctors were essential to achieve performance on par with english for morphologically complex, and typically low resource, languages. software, tutorials, and examples for constructing fst morphology are available from the finite state morphology book website. 2 we use the foma library foot_2 by @xcite , compatible with fst morphology, and available, along with some fine tutorials. both applications offer a python api, but neither is under active development. there is limited community support for foma. ## linguistic profile and resources yine (iso 639-3: pib) may be considered a morphosyntactically complex language due to its highly polysynthetic profile (mainly related to verbal structures). as noted by @xcite , arawak languages are synthetic, predominantly head marking and suffixing, with a complex verbal morphology. yine presents three open word classes: nouns, verbs, and adjectives (mostly by derivation); and four closed word classes: pronouns, adverbs, demonstratives and numerals. in this section, we will only discuss the pronominal system, and some features associated with the verbal and nominal morphology, since they are relevant to the current state of representation of yine morphology by the fst formalism. ## finite state morphology in the fst morphology formalism (figure 1a ), parallel language representations (tapes) are mapped one to the other, where by convention the upper tape corresponds to the morphological analysis and and lower tape corresponds to the word form. each level accepts (generates) valid strings in their respective tape, and either level can be transduced to corresponding (possibly multiple) strings on the other level. fsts can be stacked so that a lower or upper tape feeds into the corresponding tape of another fst. in summary: 1. words can be transduced to morphological analyses, 2. morphological analyses can be transduced to words, 3. only valid representations are accepted (generated) on either side, 4. a valid input representation may result in multiple output representations, and 5. transducers can be stacked to multiple levels. scripting for fst (see figure 1b ) includes an optional lexc language for lexicons and an expansive fst language. while lexc is a good fit for ordered concatenative morphology and is accessible for entering inventories of open category roots, it is not a natural fit for the highly agglutinative polysynthetic yine language with its relatively free order of suffixes. instead open category root inventories are edited in spreadsheets and exported via possibly multiple stacked transducers sawli.nroot.2.aln.te-te.pssd analysis sawlite form regular expressions defining fsms advanced operators on fsms and fsts meta-language for interacting with fst lexical filters lexicon alterations post-lexical filters memory filters a b c high level lexc language for lexicons python scripts to fst source files. all morphological analysis is coded in fst, consistent with efforts by @xcite @xcite for other amazonian languages. the fst language can be viewed as divided into regular expressions (defining finite state machines (fsms)) typically used for string searching or pattern matching, advanced operators on fsms or fsts, and a meta-language for interacting with fsts. regular expressions largely suffice for the analysis tape; cross-product, rewrite rule, composition, and containment advanced operators are essential for operating on fsms and fsts; define, apply, file related, and virtual stack machine related meta-commands let us construct and interact with fsts and the operating system. fst components may also be grouped functionally as lexical, post-lexical and memory filters; lexicon; and alterations (figure 1c ). filters which restrict lexical generation precede the lexicon; they serve to restrict the allowable combinations of constituent morphemes that might be generated by the lexicon. the lexicon, originates all constituent morphemes from both open and closed morpheme classes generating all possible valid (mostly) lexical sequences. sometimes it is difficult to prospectively generate only valid analyses, and so filters may be used to prevent over-generation. similarly, some problems of over-generation (e.g., duplication) are more readily solved after generation with post-lexical filters. phonological and morphophonological processing often imposes constraints on surface form realization of the morphological analyses, e.g., final vowel elision or rhotacism. such constraints are implemented as alterations of the lexical analysis. long range or discontinuous morphological relations are not readily handled by fsts, but with use of limited memory based filters, with diacritic flags, even these problems can be resolved. we chose to divide and conquer the analyzer project based on @xcite 's yine grammar structure. we define common terms, closed class morphemes, and open class roots, followed by higher level constructs expressible as single words: adjective, noun, noun phrase, verb, nominalization, predicate, and clause. in the next section on morphological analysis we will see cogent examples combining language analysis from the previous section and finite state morphology described here. ## morphological analysis we report several morphological analyses and snippets of corresponding fst code. fst is a multiuse term applying to simple definitions, regular expressions, filters, alterations, lexicon and the entire analyzer. all the terms beginning with /•/ are symbolic terms defined in file common-u.foma; their corresponding implementation specific and unimorph terms are substituted on evaluation. listing 1 shows a snippet of label definitions. ## evaluation for unit testing of noun, verb, and verb predicate analyses, we constructed forms for several distinct analyses each of 20 nouns sampled over possessor class and 20 verbs sampled over subject and object classes. diverse analyses varied possessor/subject/object person, number, and gender as well as noun or verb incorporants. while resulting derived forms were largely consistent with analyses, we discovered and corrected several cases of lexically specified vowel elision and rhotacism not covered in hanson (2010)'s grammar. for coverage on test data we sampled words matching on known root forms with 25 each of noun roots and verb roots sampled at random from a yine corpus by @xcite . this resulted in many out of vocabulary words from longer root forms than those used for selection. yet, there remained numerous other words unrecognized (not covered) by the analyzer even though sharing the expected root. so we performed a detail error analysis from a sub-sample of 63 unrecognized words to diagnose errors and make model improvements. the error analysis is reported in corrections and improvements from easy to hard are: 1. correct out of vocabulary, entry, and orthographic errors of roots on vocabulary spreadsheets. 2. correct intermediate flags and alterations for elision and rhotacism. 3. add missing suffixes and more flexible order for morpheme out of vocabulary and order errors. 4. prioritize development of non-verbal predicate, nominalizer, and verbalizer functions to address non-verbal predicate and change of category errors. cardenas and zeman (2018) obtained 78.9% average coverage over multiple domains on test data for a completed fst morphology of an amazonian polysynthetic language. our ≈15% coverage in ## conclusion we have shown our initial steps in developing noun, verb, verb predicate and pronoun categories for a morphological model of the yine language, illustrating analyses performed and fst patterns used to solve challenging problems. testing for analyzer coverage with real world data revealed several deficiencies, some expected (nominalizers, verbalizers, non-verbal predicate) and some surprises (unexpected elision, rhotacism, and missing morpheme errors). we will continue to improve the analyzer by fixing problems and adding major word categories and functions, now with added emphasis on testing with external data. goals for the analyzer include both language documentation and use as a component of natural language processing (nlp) applications such as spell checking and low resource machine translation.