An FST morphological analyzer for the Gitksan language (2021)

Abstract: This paper presents a finite-state morphological analyzer for the Gitksan language. The analyzer draws from a 1250-token Eastern dialect wordlist. It is based on finite-state technology and additionally includes two extensions which can provide analyses for out-of-vocabulary words: rules for generating predictable dialect variants, and a neural guesser component. The pre-neural analyzer, tested against interlinear-annotated texts from multiple dialects, achieves coverage of (75-81%), and maintains high precision (95-100%). The neural extension improves coverage at the cost of lowered precision.

## introduction endangered languages of the americas are typically underdocumented and underresourced. computational tools like morphological analyzers present the opportunity to speed up ongoing documentation efforts by enabling automatic and semiautomatic data analysis. this paper describes the development of a morphological analyzer for gitksan, an endangered indigenous language of western canada. the analyzer is capable of providing the base form and morphosyntactic description of inflected word forms: a word gupdiit 'they ate' is annotated gup-tr-3pl. our gitksan analyzer is based on two core documentary resources: a wordlist spanning approximately 1250 tokens, and an 18,000 token interlinear-annotated text collection. due to the scarcity of available lexical and corpus resources, we take a rule-based approach to modeling of morphology which is less dependent on large datasets than machine learning methods. our analyzer is based on finite-state technology @xcite using the foma finite-state toolkit @xcite . our work has three central goals: (1) we want to build a flexible morphological analyzer to supplement lexical and textual resources in support of language learning. such an analyzer can support learners in identifying the base-form of inflected words where the morpheme-to-word ratio might be particularly high, in a way not addressed by a traditional dictionary. it may also productively generate inflected forms of words. (2) we want to facilitate ongoing efforts to expand the aforementioned 1250 token wordlist into a broad-coverage dictionary of the gitksan language. running our analyzer on gitksan texts, we can rapidly identify word forms whose base-form has not yet been documented. an analyzer can also help automate the process of identifying sample sentences for dictionary words, the addition of which substantially increases the value of the dictionary. (3) we want to use the model to further our understanding of gitksan morphology. unanalyzeable and erroneously analyzed forms can help us identify shortcomings in our description of the morphological system and can thus feed back into the documentation effort of the language. the gitksan-speaking community recognizes two dialects: eastern (upriver) and western (downriver). our analyzer is based on resources which mainly represent the eastern dialect. consequently, our base analyzer achieves higher coverage of 71% for the eastern dialect as measured on a manually annotated test set. for the western dialect, coverage is lower at 53%. in order to improve coverage on the western variety, we explore two extensions to our analyzer. first, we implement a number of dialectal relaxation rules which model the orthographic variation between eastern and western dialects. this leads to sizable improvements in coverage for the western dialect (around 9%-points on types and 6%-points on tokens). moreover, the precision of our analyzer remains high both for the eastern and west-ern dialects even after applying dialect rules. secondly, we extend our fst morphological analyzer by adding a data-driven neural guesser which further improves coverage both for the eastern and western varieties. ## the gitksan language the gitxsan are one of the indigenous peoples of british columbia, canada. their traditional territories consist of upwards of 50,000 square kilometers of land along the skeena river in the bc northern interior. the gitksan language is the easternmost member of the tsimshianic family, which spans the entirety of the skeena and nass river watersheds to the pacific coast. today, gitksan is the most vital tsimshianic language, but is still critically endangered with an estimated 300-850 speakers @xcite . the tsimshianic family can be broadly understood as a dialect continuum, with each village along these rivers speaking somewhat differently from its neighbors up-or downstream, and the two endpoints being mutually unintelligible. the six gitxsan villages are commonly divided into two dialects: east/upriver and west/downriver. the dialects have some lexical and phonological differences, with the most prominent being a vowel shift. consider the name of the skeena river: xsan, ksan (eastern) vs ksen (western). ## related work while considering different approaches to computational modeling of gitksan morphology, finitestate morphology arose as a natural choice. at the present time, finite-state methods are quite widely applied for indigenous languages of the americas. chen and schwartz (2018 ) present a morphological analyzer for st. lawrence island / central siberian yupik for aid in language preservation and revitalization work. strunk (2020) present another analyzer for central alaskan yupik. snoek et al. (2014) present a morphological analyzer for plains cree nouns and harrigan et al. (2017) present one for plains cree verbs. littell (2018 ) build a finitestate analyzer for kwak'wala. all of the above are languages which present similar challenges to the ones encountered in the case of gitksan: word forms consisting of a large number of morphemes, both prefixing and suffixing morphology and morphophonological alternations. finite-state morphology is well-suited for dealing with these challenges. it is noteworthy that similarly to gitksan, a number of the aforementioned languages are also undergoing active documentation efforts. while we present the first morphological analyzer for gitksan which is capable of productive inflection, this is not the first electronic lexical resource for the gitksan language. @xcite present an electronic dictionary interface waldayu for endangered languages and apply it to gitksan. the model is capable of performing fuzzy dictionary search which is an important extension in the presence of orthographic variation which widely occurs in gitksan. while this represents an important development for computational lexicography for gitksan, the method cannot model productive inflection which is important particularly for language learners who might not be able to easily deduce the base-form of an inflected word @xcite . as mentioned earlier, our model can analyze inflected forms of lexemes. we extend the coverage of our finite-state analyzers by incorporating a neural morphological guesser which can be used to analyze word forms which are rejected by the finite-state analyzer. similar mechanisms have been explored for other american indigenous languages. @xcite use segmental recurrent neural networks @xcite to augment a finite-state morphological analyzer for inuktitut. 2 these jointly segment the 4 the model our morphological analyzer was designed with several considerations in mind. first, given the small amount of data at our disposal, we chose to construct a rule-based finite state transducer, built from a predefined lexicon and morphological description. the dependence of this type of analyzer on a lexicon supports one of the major goals of this project: lexical discovery from texts. words which cannot be analyzed will likely be novel lemmas that have yet to be documented. furthermore, the process of constructing a morphological description allows for the refinement of our understanding of gitksan morphology and orthographic standards. for example, there is a common post-stem rounding effect that generates variants such as jogat, jogot 'those who live'; the project helps us identify where this effect occurs. our analyzer can also later serve as a tool to explore of the behavior of less-documented constructions (e.g. distributive, partitive), as grammatical and pedagogical resources continue to be developed. our general philosophy was to take a maximalsegmentation approach to inflection and cliticization: morphemes were added individually, and interactions between morphemes (e.g. deletion) were derived through transformational rules based on morphological and phonological context. most interactions of this kind are strictly local; there are few long-distance dependencies between morphemes. the only exception to the minimal chunking rule is a specific interaction between noun-class clitics and verbal agreement: when these clitics append to verbal agreement suffixes, they either agglutinate with (6-a) or delete them (6-b) depending on whether the agreement and noun-class morpheme are associated with the same noun @xcite . that is, the conditioning factor for this alternation is syntactic, not morphophonological. realizations of gup-i-@xmath0. gubithl 'he/she ate (common noun)' b. gubihl '(common noun) ate' the available set of resources further constrained our options for the analyzer's design and our means of evaluating it. the h&r wordlist is quite small, and of only a single dialect, while the corpus for testing was multidialectal. we therefore aimed to produce a flexible analyzer able to recognize orthographic variation, to maximize the value of its small lexicon. ## discussion and conclusions the grammatically-informed fst is able to handle many of gitksan's morphological complexities with a high degree of precision, including homophony, contextual deletion, and position-flexible clitics. the fst analyzer's patchy coverage can be attributed to its small lexicon. unknown lexical items and variants comprised roughly 18% of each small dataset. notably, errors and unidentified forms in the fst analyzer signal the current limits of morphotactic descriptions and lexical documentation. the analyzer can therefore serve as a useful part of a documentary linguistic workflow to quickly and systematically identify novel lexical items and grammatical rules from texts, facilitating the expansion of lexical resources. it can also be used as a pedagogical tool to identify word stems in running text, or to generate morphological exer-cises for language learners. the neural system, with its expanded coverage, can serve as part of a feedback system with a human in the loop, informing future iterations of the annotation process. while its precision is lower than the fst, it can still inform annotators on words that the fst does not analyze. newly-annotated data can then be used to enlarge the fst coverage.