Linguist vs. Machine: Rapid Development of Finite-State Morphological Grammars (2020)

Abstract: Sequence-to-sequence models have proven to be highly successful in learning morphological inflection from examples as the series of SIGMORPHON/CoNLL shared tasks have shown. It is usually assumed, however, that a linguist working with inflectional examples could in principle develop a gold standard-level morphological analyzer and generator that would surpass a trained neural network model in accuracy of predictions, but that it may require significant amounts of human labor. In this paper, we discuss an experiment where a group of people with some linguistic training develop 25+ grammars as part of the shared task and weigh the cost/benefit ratio of developing grammars by hand. We also present tools that can help linguists triage difficult complex morphophonological phenomena within a language and hypothesize inflectional class membership. We conclude that a significant development effort by trained linguists to analyze and model morphophonological patterns are required in order to surpass the accuracy of neural models.

## introduction hand-written grammars for modeling derivational and inflectional morphology have long been seen as the gold standard for incorporating a word inflection aware component into nlp systems. however, the recent successes of sequence-to-sequence (seq2seq) models in learning morphological patterns, as seen in multiple shared tasks that address the topic @xcite @xcite , have raised the question whether there is any advantage in developing handwritten grammars for performance reasons. this question has special relevance with regard to lowresource languages when there is a desire to quickly develop fundamental nlp resources such as a morphological analyzer and generator with minimal resource expenditure @xcite . it is clear that there is a need for hand-written morphological grammars, even if neural network models approach the performance of carefully hand-crafted morphologies. normative and prescriptive language models, such as those needed by language academies in many countries-e.g. rae in spain, académie franc ¸aise in france, or the council of the cherokee nation in the u.s.would need to rely on explicitly designed models for providing guidance in word inflection, spelling rules, and orthography if they were to be implemented computationally. currently, neural models trained on examples provide no verifiable guarantees that certain prescriptive phenomena have been learned by a trained model and can be reliably used. in this paper foot_0 we document an experiment where a number of morphological grammars were handwritten by a group of 19 students enrolled in the class "ling 7565-computational phonology & morphology" at the university of colorado, each student having training in either computer science or linguistics, and some previous training in writing finite-state morphological grammars. the languages were chosen from the 2020 sigmor-phon shared task 0 @xcite , and the grammars were designed so as to be able to inflect unseen forms. the design was also such that the grammars were able to function as "guessers" and inflect lexemes never seen in the training data. ## finite-state grammars finite-state transducer (fst) solutions have long been the foremost paradigm in which to develop linguistically informed large-scale morphological grammars @xcite @xcite . the availability of a variety of tools @xcite ; @xcite ; @xcite inter alia) has also supported this mode of development, and by now hundreds of lan-guages have grammars developed by linguists in this paradigm. the usual approach to developing morphological analyzers is to model the mapping from a lemma (citation form) and a morphosyntactic description (msd) into an inflected form (target form) as a twostep process. the first step maps the lemma+msd into an intermediate form that represents a combination of canonical morpheme representations, while the second step employs a cascade of transducers which handle morphophonological alternations. it is customary to handle inflectional classes by explicitly dividing lemmas into groups in the first step so that correct morphemes are chosen for each lemma. analyzers built in such a way generally are not capable of inflecting lemmas that are not explicitly encoded in a lexicon. however, it is common to integrate an additional "guesser" component that can handle any valid lemma in a language, and pass it through the relevant morphophonological component only @xcite . basic finite-state calculus is then used to construct a single fst that "overrides" outputs from the guesser whenever a known lexeme is inflected, so conflicting outputs are avoided. the basic design is illustrated in figure 1 . ## approach all of the grammars were built with the foma finitestate tool @xcite . before grammar writing commenced, the participants were urged to spend roughly 1 hour in groups of 3 to quickly analyze all the languages in the development and surprise groups as follows: through the above process, a number of languages were selected as the primary targets, and development was launched for some 40 languages in total-roughly 20 for the development languages and a similar number for the surprise languages, as they were published. in the end, the output of 25 languages was submitted to the shared task. the criterion for actually submitting a language was that the grammar was mature enough, judged by examining whether accuracy on the development set was within 5% of the neural baseline models @xcite provided by the organizers. ## tools as mentioned above, a number of tools for the support of rapid grammar writing were also developed. these included the tools to reconstruct the partial inflection tables from the data and various analysis tools for accuracy and error reporting. apart from that, a separate tool for inflection table clustering and a non-neural tool for hypothesizing forms for missing slots in paradigms were also developed. this latter tools' output was also submitted as a second system (cu-7565-02) to the shared task for nearly all languages. these two tools were more involved and are discussed in detail below. ## results the results for the hand-written grammars (cu-7565-01) and the non-neural paradigm completion model (cu-7565-02) are given in the non-neural paradigm completion model (cu-7565-02), which was submitted for nearly all 90 languages performed reasonably well, and is to our knowledge the best-performing nonneural model available for morphological inflection. never outperforming the strongest neural models; it nevertheless represents a strong improvement over the baseline non-neural model provided by the organizers. additionally, it provides another tool to quickly see reasonable hypotheses for missing forms in inflection tables. ## conclusion we have done a preliminary investigation in pitting neural inflection models against more traditional hand-written grammars, designed by nonnaive grammar developers with some training in the field of linguistics and computational modeling. the results point to two main directions. first, it is very difficult in many cases to outperform a state-of-the-art neural network model without significant development effort and attention to nuanced morphophonological patterns. indeed, some data sets in the task were very simple, and in such cases, it is quite trivial to develop a highaccuracy grammar. this advantage is somewhat nullified by the apparent ability of neural seq2seq models to also model such morphologies with high accuracy, despite little data. the second observation is the following: for languages where the group was able to significantly outperform neural models (such as tagalog and ingrian), success did not come cheaply. we estimate that for any language with high morphophonological complexity and a variety of inflectional classes, possibly hundreds of hours of development effort is required even by a trained linguist to surpass the performance of a current state-of-the-art seq2seq model. but it is also precisely in this latter case of high-complexity languages where linguists can still prevail with a margin.