Linear Discriminative Learning: a competitive non-neural baseline for morphological inflection (2023)

Abstract: This paper presents our submission to the SIGMORPHON 2023 task 2 of Cognitively Plausible Morphophonological Generalization in Korean. We implemented both Linear Discriminative Learning and Transformer models and found that the Linear Discriminative Learning model trained on a combination of corpus and experimental data showed the best performance with the overall accuracy of around 83%. We found that the best model must be trained on both corpus data and the experimental data of one particular participant. Our examination of speaker-variability and speaker-specific information did not explain why a particular participant combined well with the corpus data. We recommend Linear Discriminative Learning models as a future non-neural baseline system, owning to its training speed, accuracy, model interpretability and cognitive plausibility. In order to improve the model performance, we suggest using bigger data and/or performing data augmentation and incorporating speaker- and item-specifics considerably.

## introduction there has been a heated debate on whether human language users generate language in a generative manner (e.g., @xcite ) or an output-oriented manner (e.g., @xcite ). in accordance with the theoretical stance, computational models have been proposed. the generative approach is essentially rule-based at an abstract level. the minimal generalisation learner (mgl) by @xcite is † these authors contributed equally to this work. senior and corresponding author: kevin tang (kevin.tang@hhu.de) a traditional, symbolic rule learner. more recent rule-based computational approaches include @xcite and @xcite . with the availability of large corpus data, outputoriented models have become widely popular. output-oriented models can be rule-based or endto-end. the former includes @xcite and @xcite ; in the former, a search procedure is implemented on a set of candidates, or outputs, in order to find the surface form that is most compatible with the underlying representation. on the other hand, output-oriented models can be rule-free. for instance, @xcite showcased a recurrent deep learning model to predict paradigm forms. similarly, @xcite proposed a encoder-decoder network architecture to model linguistic phenomena. both approaches have their own advantages and disadvantages. however, in terms of model performance, output-oriented models surpass generative ones presumably due to difficulty of incorporating many other variations in conversation. nevertheless, output-oriented models are not panacea. they are not as cognitively motivated, thus making them less appealing for cognitive research. deep learning-based models, in particular, show great performance, but they lack interpretability. these shortcomings necessitate a hybrid model, which is i) cognitively motivated, ii) more agnostic than generative models, and iii) more transparent than deep learning-based models. recently, @xcite proposed linear discriminative learning (ldl), part of the discriminative lexicon @xcite . as the model follows the rescorla-wagner rule and widrow-hoff rules, some insight into human cognition can be obtained through model imple-mentation. moreover, as it implements a linear mapping between form and meaning in simple two layers without any hidden layer, ldl features higher interpretability and embraces linguistic engineering. considering these advantages, an ldl model was chosen as the main model for the sigmorphon 2023 shared task 2, which aims to generalize morphological inflections in korean. a transformer model, which has been state-of-the-art models for various nlp tasks, was also implemented for comparison. the code and data are available here: https://github.com/ hhuslamlab/sigmorphon2023 2 related work while neural-based systems typically dominate sigmorphon challenges, perhaps because they generally perform well in morphological inflection tasks, their limitations can be better examined using wug testing. for instance, @xcite examined the ability of modern encoder-decoder (ed) architectures to inflect german plurals and concluded that ed does not show humanlike variability as shown in wug data. in fact, recent sigmorphon challenges involve learning from corpora that better represent the actual linguistic input of children (such as child-directed speech) and evaluating on phonetically-transcribed spoken production by children or adults in corpora or in experiments. for instance, the sigmorphon 2021 shared task 0 part 2 was to predict the judgement ratings of wug words @xcite as opposed to using real words held-out from the training data as test data. similarly, the sigmorphon 2022 challenge involved computational modeling of the data drawn from corpora of child-directed speech and evaluation on children's learning trajectories and erroneous productions @xcite . turning to studies from the field of laboratory phonology, there is a long history of training models on corpora to learn specific aspects of morphophonological grammar and evaluating their productivity with experimental data (wug-test and acceptability judgement). for instance, jun (2010)'s study on stem-final obstruent variation in korean trained a model with multiple stocastic rules @xcite 's paradigm learning model) in which the acquisition of morphology is based on the distributional pattern of the learning data, using the sejong corpus, and evaluated on acceptabil-ity judgement data. related to the linguistic phenomenon in this study, @xcite conducted a computational modeling of inflected forms of korean verbs using the minimal generalization learner algorithm @xcite and evaluated the model's performance with attested child errors and historical changes. finally, there is a growing number of morphological inflection studies that use the linear discriminative learning model (which will be introduced later) to train on corpus data and evaluate on experimental data @xcite @xcite @xcite and they typical yielded relatively high performance (compared to traditional, symbolic rule learner) while being easy to interpret and cognitively motivated. ## task and evaluation details we challenge the shared task 2 cognitively plausible morphophonological generalization in korean. the aim of this task is to predict human responses to a generalization task (wug-test), considering high-frequency, low-frequency, and pseudoword items. this implies that human responses may vary depending on the word frequency and familiarity. the phonological phenomenon to be tested through this task is korean post-obstruent tensification (henceforth, pot). in korean, when a lenis consonant in the coda is followed by another obstruent, it can be tensified. however, when a consonant cluster occurs in the coda position, it undergoes consonant simplification (henceforth, cs) before pot if the following segment is either an obstruent or a sonorant. on the other hand, neither cs nor pot does occur when the following segment is a vowel. depending on the type of the deleted consonant, variation can occur, which can be affected by such speaker-and item-specific features as language familiarity and frequency. in this regard, the main task can be rephrased as predicting features as completely as possible in accordance with those in the answers. for this task, both corpus and experimental data are provided, which include variation patterns. models are to be evaluated on the accuracy of the prediction of the feature vectors given the corresponding features in the answers from unseen participants. ## data both corpus and experimental data are provided for this task. the national institute of korean language (nikl) korean dialogue corpus (nikl, 2021) is provided as the main corpus data. all the word tokens affixed with -lc verbs, except for -lh final stems, are provided after manual annotation by the organizers. they further are categorized as lc+obstruent, lc+sonorant, and lc+vowel, depending on the type of the consequent segment. lc+obstruent and lc+sonorant data each include target words with morphological boundaries and produced words with syllable breaks both in a romanized form and korean orthography. whether target words undergo pot is also provided, as well as such features as obstruent deletion and lateral deletion pertinent to cs and pot. on the other hand, lc+vowel data only provide target words with morphological boundaries both in a romanized form and korean orthography. they do not include produced word information as this condition is not subject to pot and cs; thus, all the feature values of obstruent deletion and lateral deletion are labeled as 0 with the pot value being labeled as 0. whether the target is lateralizated or nasalized is marked only in the lc+sonorant data while labeled as na in the others. the number of tokens in lc+obstruent data is 876, that of tokens in lc+sonorant data is 95, and that of tokens in lc+vowel data is 2,525 with 514 types-1,485 if frequency information in lc+vowel data is ignored. thus, the total number of tokens in the nikl data is 3,496. in addition to the adult corpus, some part of a child spontaneous speech corpus, the ko corpus @xcite , is provided. as with the nikl data, the ko corpus provides target words and their production but solely in a romanized form. the pot value with the lateralization and nasalization feature values are labeled. as it does not include any lc verbs followed by a sonorant, both nasalization and lateralization are not applicable. a total of 336 tokens are provided. in the case of experimental data, the experimental responses of 12 participants, in addition to 4 participants for the development data, are provided. they include what are included in the corpus data with the experimental specifics: the trial number, the trial id, the subject id, option, language familiarity, and word frequency. a total of 2,843 tokens are provided. ## system description the main task is to accommodate as many variation patterns as possible. we navigated through all the corpus and experimental data, in which process we found inconsistencies in transcription in the data. in particular, the ko corpus includes items transcribed in a very detailed manner, including phonological processes, such as deletion and insertion, other than those pertinent to the task. we also found that the target item does not always feature one-to-one mapping, but more-than-one mapping. based on the observations above, data preprocessing was of primary importance and was the most time-consuming component. we manually corrected the ko corpus and automatically unified the transcription style. we then selected models adequate to this task. considering the nature of the shared task that investigates morphological variations with both corpus and experimental data and the time constraint, ldl was chosen as the main model, along with a transformer model that has demonstrated great performance in nlp. for each of the two modelling approaches we conducted two studies: study 1 experimented with systems that train only on the corpus data and/or only on the experimental data; study 2 used the insights from study 1, and experimented with systems that train on both the corpus and the experimental data. ## variability: items and participants to examine the variability of the phenomenon, shannon entropy (base 2) (shannon, 1948) was used to quantify how variable are the items in the experimental data and how variable are the participants. in this study, we considered sixteen possible combinations of the five phonological processes (therefore sixteen events in entropy's term) (see appendix c). with sixteen combinations, the highest possible entropy value is 4 which means each combination has a probability of 1/16 indicating a high level of variability, and the lowest possible entropy is 0 which means there is only one attested combination indicating no variability. for detailed analyses, see appendix b. first, we computed the by-item entropy values by computing the proportion of the sixteen response combinations using the sixteen participants (training and development). the 180 ex-perimental items have a mean entropy of 0.584. pseudoword items have the highest mean entropy (0.612), followed by high-frequency items (0.596) and low-frequency items (0.544). these entropy values suggest that the experimental items in general have low variability and unsurprisingly the pseudoword items were particularly variable compared to the real words. however, these differences in entropy values were not statistically significant (@xmath0.3842). second, we computed by-participant entropy values by computing the proportion of the sixteen response combinations. across all the experimental items, the sixteen participants have a mean entropy of 2.143. participants show the lowest mean entropy with the high-frequency items (2.049) , followed by low-frequency items (2.111) and pseudoword items (2.112). however, these differences in entropy values were not statistically significant (@xmath1.2542). our 'best' participant 597515 has an entropy of 2.192 across all items, 2.176 for highfrequency, 2.154 for low-frequency and 2.140 for pseudoword items, with all values similar to their means. therefore, the participant's superiority is not purely due to their responses being more variable. ## discussion and conclusion we demonstrated that ldl is capable of modelling morphological inflection trained on limited corpus and experimental data. its performance is competitive to that of the transformer model that we experimented with. past sigmorphon shared tasks (2017-2022) with a focus on morphological inflection have generally received more neural-based systems than non-neural ones and found that neuralbased ones tend to be superior @xcite @xcite @xcite @xcite . amongst the submitted non-neural systems, ldl has never been utilized. our study cannot conclude that ldl is superior to the transformer architecture as the latter was not fully optimized. however, it has great potential to serve as a non-neural baseline system for future shared tasks as well as allowing researchers to conduct rapid experiments, because of its architecture simplicity, performance (with accuracies from 59% to 99% in a range of languages, e.g. @xcite @xcite @xcite and speed (in our study one model required on average 35 seconds of cpu processing on an i7-9750h 2.60 ghz system with 32 gb memory). our study found that training on the corpus data alone was insufficient and that our models require at least one participant's experimental data in order to inflect the experimental items well. however, from an ecological perspective, a model should only be trained on the corpus data (nikl and ko), excluding the experimental data, as the corpus data serve to represent the participants' actual linguistic input. the corpus data we have are likely unrepresentative of the actual linguistic input of our participants. firstly, the verbs were not embedded in an utterance, and even if the full utterances were used the overall amount of data would still be small with only 53,000 words from the ko corpus, and 900,000 phrases from the nikl corpus. based on spoken speech input alone, @xcite estimated that for american english, the total input from social interactions (in a dialogue) would be equal to 11.688 million word tokens per year and a 20-year-old would have been exposed to about 234 million word tokens. using a much larger speechlike or transcribed corpus such as subtlex-kr (tang and de chene, 2014) (90 million eojeols) is a promising approach for examining morphological inflection patterns (see de chene (2014) on regularisation in korean noun inflection). our item variability analyses suggest that the three item types (high-, low-frequency and pseudowords) are not particularly different in their variability. this might be reflecting how the ldl model reported in section 5.3 performed similarly with them (high: 83.58%, low: 83.56%, and pseudowords: 82.84%). however, the best transfomer model was sensitive to item types yielding a higher accuracy for the high-frequency items (77.7%), than the low-frequency (64.4%) and pseudoword (65.38%) items. our attempts in understanding why the 'best' participant was the best in predicting individual participants' productions were not successful. response variability was unable to explain why our 'best' participant was the best, as it has neither high nor low in variability compared to the other 15 participants. our regression analyses predicting individual participant accuracies using the participants' demographics was inconclusive. while one may assume that the ldl prediction results should improve when one predicts speakers of similar back-grounds, the nonetheless satisfying ldl prediction results suggest that demographics-matching was not needed. overall, our results suggest that ldl is suitable for tasks such as the one at hand.