Not Quite There Yet: Combining Analogical Patterns and Encoder-Decoder Networks for Cognitively Plausible Inflection (2021)

Abstract: The paper presents four models submitted to Part 2 of the SIGMORPHON 2021 Shared Task 0, which aims at replicating human judgements on the inflection of nonce lexemes. Our goal is to explore the usefulness of combining pre-compiled analogical patterns with an encoder-decoder architecture. Two models are designed using such patterns either in the input or the output of the network. Two extra models controlled for the role of raw similarity of nonce inflected forms to existing inflected forms in the same paradigm cell, and the role of the type frequency of analogical patterns. Our strategy is entirely endogenous in the sense that the models appealing solely to the data provided by the SIGMORPHON organisers, without using external resources. Our model 2 ranks second among all submitted systems, suggesting that the inclusion of analogical patterns in the network architecture is useful in mimicking speakers’ predictions.

## introduction psycho-computational experiments deal with the capability of the computational models of language to capture and mimic the behavioural responses of speakers exposed to the same data stimuli. in phonology, and in morphology, a large number of computational models (both symbolic and subsymbolic) have been tested on various experimental data in order to evaluate their capacity to simulate linguistic behavior @xcite @xcite @xcite . more recently, @xcite argue that a neural encoder-decoder network (ed) can perform morphological inflection tasks in a cognitively valid manner. in particular, the authors claim that, in a wug test protocol, ed's outputs significantly correlate with human judgements for nonce verbs supporting the assumption that the model learns representations of specific knowledge and involves cognitive mechanisms that are known to underlie language processing in the speakers. however, whether and how such models are able to mimic the human behaviour of subjects exposed to the same stimuli is still an open question @xcite . part 2 of shared task 0 addresses this same issue. more specifically, it adopts the experimental approach of @xcite . the task is to design models which predict the inflected forms of a set of nonce verbs in a given language and that have output scores of the predictions that correlate with human judgements. in this paper we report on a series of experiments that address the shared task by exploring whether pre-learned formal analogies between strings can be usefully combined with an ed architecture to alleviate some limitations of the application of an ed architecture to raw forms. we present two types of models using analogical patterns in the input (m1) vs. in the output (m2) of an ed network, and compare their performance to that of two baselines that focus respectively on the phonotactic typicality of outputs (m3) and the type frequency of alternations (m4). we report good performance for the m2 architecture and quite poor performance for m1. in section 2, we shortly present the data provided by the organisers. section 3 focuses on the analogical patterns we use in our models. the models architecture, the training parameters and the results are reported in section 4. we then discuss the results in section 5 and draw conclusions (section 6). ## data and goal the linguistic data provided by the organisers are inflected verb forms in english (eng), german (deu) and dutch (nld). for each language, four datasets are released: (a) a training dataset, (b) a development dataset of attested verb forms, (c) a development dataset of wug forms which includes human judgements, and (d) a test dataset of wug forms without the human judgements. the goal of the shared task is to assign a score to each wug form in (d) that correlates as closely as possible with the human judgements (see section 4.2 for more details on the evaluation process). the entries of the datasets include lemma/form pair and a unimorph @xcite tag (ut) specifying the part of speech and the paradigm cell of the form. the pairs are provided as written forms (orthographically) in datasets (a) and (b) and in ipa (phonologically) in all four datasets. ## analogical patterns three of our model architectures rely on alternation patterns (aps) describing the formal relationship between two word-forms. an ap is a pairing of two word-forms patterns (wps) with shared variables over substrings which represent word parts that are common between the two forms. for example, the two german word-forms anspi:l@n 'to allude to' and anspi:l@nt 'alluded to' are related by the pattern (anx@n, anx@nt) where the variable x represents spi:l. the number of different aps satisfied by a pair of forms is typically large. for instance, there are 256 (2 8 ) distinct patterns relating anspi:l@n and an-spi:l@nt, some of which are shown in (1), where italic capital letters represent variables over strings. henceforth we will note patterns using the '+' symbol as a general notation for variables, and rely implicitly on order to match variables in alternations. hence e.g. the ap (xny pzlt n, xny pzlt nt) will be noted +n+p+l+n/+n+p+l+nt. most of the patterns in (1) are of little morphological interest. this is in particular the case of the trivial alternation pattern (tap) which just records the two strings without making any generalization over common elements. a broad alternation pattern (bap) is an optimal pattern that can be inferred by pairwise alignment of the two forms, without taking into account the situation in the rest of the paradigm. in principle there can be more than one bap for a pair of form, although that rarely happens in practice. this type of pattern is of crucial interest to the study of the implicative structure of paradigms @xcite and the induction of inflection classes @xcite , but does not lead to the identification of affixes familiar from typical grammatical descriptions. for that purpose, multiple alignments across the paradigms are necessary @xcite , and lead to what we call a fine alternation pattern (fap): here @n is the infinitive suffix and @nt is the present participle suffix. the crucial intuition behind the determination of faps is that they identify recurrent partials @xcite across both paradigms and lexemes. for instance, the fap in (1) is motivated by the fact that the substrings an and spi:l are shared across all pairs of paradigm cells of anspielen (2), while the substrings @n and @nt recur in many (infinitive, present participle) pairs across lexemes (3). in this paper, we rely on an algorithm for inferring baps and faps initially developed to create glawinette @xcite . glawinette is a french derivational lexicon created from the definitions of the glawi machine readable dictionary @xcite . glawinette provides a description of derivational morphology by means of morphological families and derivational series; it is part of an effort aiming at the design of derivational paradigms. baps and faps have been adapted to the datasets of the current task, analogizing inflectional paradigms to derivational families and pairs of inflectional paradigm cells to derivational series. for instance, (4) presents an excerpt of an inflectional series in the inflectional paradigm of the verb anspielen that realizes the features v;nfin and v.ptcp;pst. in turn, this series yields two series of word-forms, the ones in the left column and the ones in the right column. ## combining analogical patterns and encoder-decoder networks early work on connectionist models of the acquisition of morphology involved pattern associators that learn relationships between a base lexical form (i.e. the lemma) and a target form (i.e. the inflected form). for example, @xcite propose a simulation of how english past tense is learned. they focus on pairs of verb forms like go-went and walk-walked and consider that morphological learning is a gradual process which includes an intermediate phases of "over-regularization" (where the past form of go is goed instead of went). this yields the wellknown"u-shape" curves observed in the developmental phases of morphological competence in children. more recently, models based on deep learning architectures have been used @xcite and in particular sequence-to-sequence models able to predict one form of a lexeme from another @xcite . these approaches are based on the assumption that the morphological learning can reduce to a simple mapping between a base form and an inflected one. generalizations over similar mappings (e.g. love-loved,walk-walked vs. sing-sang, ringrang) are learned from the dependences between the phonemes in sequences. the aps presented in section 3 provide a description complementary to the lemma-form mapping in which analogical regularities may be locally to a single pair of forms (baps) or globally from the entire lexicon (faps). these paradigmatic analogies emerge when the forms are contrasted with all other forms of their lexeme and the other forms that occupy the same cell in the paradigm @xcite @xcite . the models we designed for the task combine the capacity of the sequence-to-sequence models to learn the regularities present in strings of phonemes with the alternation patterns acquired from the paradigms, in order to predict native speaker responses in a wug test. ## discussion the performance of our four models suggest the following observations. first, m2 outperforms our three other models for all three languages, and also ranked second of all systems submitted to the shared task. second, there is a striking difference in performance between m2 and m1, which had an similar architecture, but performed very poorlyworse than our baseline m4 model, and second to last of all systems submitted to the shared task. although more experiments are needed to conclude on this point, we conjecture that the better performance of m2 might be due to the fact that it abstracts away from the question of predicting the shape of the stem in the output, but focuses instead on that part of the inflected form that is not to be found in the input. this seems to match intuitions about human behavior: when dealing with inflections, speakers may have a hard time applying the right pattern, but they never have a hard time remembering what the stem looks like, even if it is phonotactically unusual (see @xcite for a psycho-computational study). the other surprising result is that m4, which was intended as a crude baseline, did surprisingly well on the english and german data, although it performed very poorly on dutch. this is interest-ingly complementary to the performance of m3, which did surprisingly well on dutch but poorly on german. as m3 is entirely focused on phonotactic similarity while m4 is focused on the frequency of alternations, this suggests that the three inflection systems (to the extent that they are faithfully represented by the datasets) raise different kinds of challenges to speakers. to better assess the quality of m2, we examined how well it statistically correlates with human performance in @xcite experiments on islands of reliability (ior) in regular and irregular inflection in english. albright and hayes are trying to establish that speakers rely on structured linguistic knowledge as encoded in their minimal generalization learner (mgl, @xcite rather than pure analogy when inflecting novel words. to establish this, they collected both productions of human participants asked to inflect a novel word, and judgments on pairs of word-forms. they show that the mgl leads to a better correlation with human results than a purely analogical system based on @xcite (nos in the table below). as ## conclusion at the time of writing, we do not have the descriptions of the other systems that were submitted to the shared task. as a result, we are not able to identify the reasons for the good and not so good performance of the four systems we submitted. the objective of our participation was to test different hypotheses. the main one is the relatively low importance of stems when predicting the acceptability of wug forms, as evidenced by the good performance of the m2 model, which which only predicts the fap of the inflected form. therefore, m2 is output-oriented in the sense that the properties that characterize the input, i.e. the lemma, are not used during training. m1 and m2 models are able to predict inflected forms and fap 2 patterns for any ut in the training set while m3 models are specialized on a single ut. in future work, we plan to develop specialized versions of m1 and m2 in order to estimate the importance of the tested inflectional series (i.e. of the set of form pairs with the same uts as the entries in test set) with respect to the entire training set. we further plan to test our models on more complete datasets in which the inflected forms could be predicted from other forms than the lemma, but also jointly from several forms of the lexeme @xcite .