Morphology is not just a naive B ayes – U ni M elb Submission to SIGMORPHON 2022 ST on Morphological Inflection (2022)

Abstract: The paper describes the Flexica team’s submission to the SIGMORPHON 2022 Shared Task 1 Part 1: Typologically Diverse Morphological Inflection. Our team submitted a nonneural system that extracted transformation patterns from alignments between a lemma and inflected forms. For each inflection category, we chose a pattern based on its abstractness score. The system outperformed the non-neural baseline, the extracted patterns covered a substantial part of possible inflections. However, we discovered that such score that does not account for all possible combinations of string segments as well as morphosyntactic features is not sufficient for a certain proportion of inflection cases.

## introduction previous years' shared tasks on morphological reinflection demonstrated superior performance across a variety of typologically diverse languages, especially in high-resource setting @xcite @xcite @xcite . still, in low-resource setting and languages with limited resources in which paradigms were only partially represented the accuracy numbers were much less optimistic @xcite . recently, @xcite experimented with the 2020 shared task data splitting it by lemmas and demonstrated the 30% accuracy drop on average among top-3 top ranked systems in that year's shared task. this motivated organizers of this year's shared task to focus on various aspects of morphological generalisation and conduct controlled experiments to evaluate systems' ability to predict inflected forms for unseen lemmas and morphosyntactic feature combinations. in this paper, we describe a modification of our earlier model, flexica @xcite , that has been participated in the 2020 shared task @xcite . 1 we provide a summary of its modified version where we attempted to improve its pattern-based generalization ability. we added ability to reuse word forms observed at different combinations of grammatical tags. also, we improved scoring mechanism to enable better fitting to rule-and-exception hierarchy which typically presents in a language, and to reduce noise in pattern selection. in addition, the training data sizes vary from 700 training instances in the small (low-resource) setting to 7,000 instance in the large (high-resource) setting. for some under-resourced languages the large setting contained fewer samples. ## baseline systems as in previous years' shared tasks, two types of baseline systems were provided: neural and nonneural. the non-neural baseline aligns extracts suffixes and prefixes based on lemma-form alignments, later associating them with corresponding morpholosyntatic features @xcite . as the neural baseline, organizers provided a character-level adaptation of transformer @xcite . ## evaluation the systems submitted to the shared task were evaluated in terms of test accuracy between predicted and gold forms. besides the overall accuracy, four categories were distinguished in the analytic data provided by organizers. depending on whether a test sample lemma has been seen in the training set, and whether an exact tag combination ("feature") has been seen in the training set, a test sample might fall into one of the following four categories: "lemma overlap", "feature overlap", "neither overlap", or "both overlap". ## results tables 1 and 2 present accuracy across all the shared task's languages measured for the small and large settings, respectively. for flexica, the column "b" stands for the basic option (without representative bonus), while the column "r" stands for the option with representative bonus. the official submission accuracy numbers are shown in the "sb." column. also, accuracy results for the nonneural and neural baselines ("bl") and best results across neural systems submitted to the shared task ("neural"/"max"), are presented for the reference. we also explored some modifications to pattern scoring, but they did not affect performance much. in particular, we tried the following options: as exact tag combinations were significantly sparse in training and test sets, the majority of mispredictions can be attributed to failures to inference tag interchangeability. indeed, in most cases of misprediction a correct transformation was available in the learnt model, but it deemed to be irrelevant due to low "similarity" between the learnt tag combination and the target one. the "av." column in tables 1 and 2 shows the percentage of test samples where a correct transformation was available for the model. it tells the upper bound of accuracy that our system would have if the pattern selection mechanism worked perfectly. ## discussion the system we explored in this paper relies on two simple hypotheses. according to the first one, a choice of inflection paradigm in most cases may be associated with some distinctive subsequence of characters in a lemma. the second hypothesis claims existence of a hierarchy of rules and exceptions in most languages, where each exception domain is fenced by a more concrete character pattern than one associated with an embracing general inflection rule. we note that our current approach only admits a very restrictive meaning for such a "concreteness", namely, the number of concrete characters in a template. due to this substantial limitation, we only consider an approximate split of rule-specific domains. while the analysis of predictions suggests this approach is generally reasonable, the distinguishing of relevant patterns from noise is challenging. certain information-based criteria such as entropy, cross-entropy and the like did not work, mainly due to specific patterns being sparsely distributed in the dataset (especially small ones), so that majority of highly concrete patterns peaked the distribution of inflection transformations. on the other hand, many relevant generic patterns demonstrate rather disperse distributions due to numerous exceptions. as a result, it is not possible to easily link the entropy to the relevancy. we intentionally avoided imposing extra biases toward "known" common language rules in order to focus our exploration on the system's learning capability itself. unfortunately, we have not yet found universal enough criteria to assess pattern relevance against inflection rules, so in this aspect the system should be considered as a work in progress. we attempted "promotion" of one maximally abstract pattern per training sample, that match the given sample and does not contradict any other observed samples. the underlying hypothesis was that every inflection paradigm is probably justified by a single "cause", where a "cause" in our restricted context stands for a distinct character pattern for a lemma. therefore, it should be reasonable to restrict prediction selection to those transformation patterns which were proven to be correctly representing at least one training sample in the most generic way. however, our experiments disproved such an approach, because, as we already said above, relevance criteria based on distribution purity are fundamentally flawed. our system operates at character level without considering more generic classes of sub-patterns. however, it did not seem to be a significant issue in most languages. in other words, patterns needed for correct inflection have usually been successfully learnt in most languages (still, non necessarily with the same grammatical tag). however, there are numerous languages where correct patterns cannot be found for a large fraction of examples; this severely jeopardised the respective prediction rates. besides the "genuinely" high morphological complexity of languages such as veps, there also occurred some "technical" reasons for the pattern match missing, such as non-standardized scripting of spoken languages (pomak, evenki). it is our system's lack of a mechanism for the affix concatenation which was responsible for inferior results observed in agglutinative languages like turkish of hungarian, especially in their low-resource settings. in the 2022 shared task, we faced a new challenge of extreme sparsity of grammatical tag combinations. a separate model per learnt tag combination does not work in such an environment. we allowed using transformation patterns observed at grammatical tag combinations different from a requested one, with a score penalty proportional to the number of different "atomic" tags (morphosyntatic features). from the inflection perspective, many grammatical tags are not as significant for a correct prediction as others are. this inspired us to use variable penalty per tag substitutions, which was proportional to a log-likelihood of observing the same transformation regardless whether a given tag is present, as measured over all learn transformation patterns, without considering other tags. for instance, in polish, the animacy does not affect inflection paradigms much, and ignoring it would significantly increase the average accuracy of inference. however, to our surprise, according to the likelihood, some case tag substitutions occurred to be better candidates for being ignored. for instance, the dative and the instrumental cases produce same forms for a majority of polish feminine nouns, therefore our predictor frequently chooses @xmath0, which is usually incorrect beyond the feminitive (instead of correct @xmath1. thus, such bayesian approach, that considers tags independently, even failed to outperform a simplistic technique based on the "edit distance" between tag combinations. we did not yet consider more complex sub-combinations of tags, still the results definitely suggest one to do that way. an excessive number of generated patterns is another challenge which yet needs to be addressed. currently, our system unrolls all the combinations of concrete characters in lemma patterns until ultimately discriminative ones are found over a training set. this leads to huge proliferation of noisy patterns of no extra value. in practice, this fact prevents the system from considering longer subsequences of concrete characters where those subsequences could really help to delimit paradigm domains. summarizing our impressions from the experiments, we suggest that the system is primarily interesting as it prototypes a simple but efficient approach to the conversion of a sequence-to-sequence task into a "plain" classification task. in this view, further enhancements of the system may be broken into two separate directions. the first one concerns the pattern matching mechanism which would become less consuming, more generalized, based on incrementally collected "cues" (and, in such a way, borrowing features of the "soft attention"). another direction, which is less specific, would be an exploration of better classification models to be used. also, the principally optimistic results obtained in our experiments inspire us to attempt expanding the proposed multi-pattern approach into other sequence-to-sequence tasks beyond the reinflection one. ## conclusion we developed a non-neural system for morphological inflection. we submitted it to the sigmor-phon 2022 shared task 1, part 1. the system outperformed the non-neural baseline, still we discovered a fundamental insufficiency of simplistic approaches that rely on observed probabilities of particular transformation patterns.