How do we get there? Evaluating transformer neural networks as cognitive models for E nglish past tense inflection (2022)

Abstract: There is an ongoing debate of whether neural network can grasp the quasi-regularities in languages like humans. In a typical quasi-regularity task, English past tense inflections, the neural network model has long been criticized that it learns only to generalize the most frequent pattern, but not the regular pattern, thus can not learn the abstract categories of regular and irregular and is dissimilar to human performance. In this work, we train a set of transformer models with different settings to examine their behavior on this task. The models achieved high accuracy on unseen regular verbs and some accuracy on unseen irregular verbs. The models’ performance on the regulars are heavily affected by type frequency and ratio but not token frequency and ratio, and vice versa for the irregulars. The different behaviors on the regulars and irregulars suggest that the models have some degree of symbolic learning on the regularity of the verbs. In addition, the models are weakly correlated with human behavior on nonce verbs. Although the transformer model exhibits some level of learning on the abstract category of verb regularity, its performance does not fit human data well suggesting that it might not be a good cognitive model.

## introduction many aspects of language can be characterized as quasi-regular: the relationship between inputs and outputs is systematic but allow many exceptions. english past tense inflection exhibits such quasi-regularity that the regular verbs follow the '-ed' rule (help -helped) and the irregular forms consist of a variety of changes such as changing vowel (sing -sang). there has been heated debate about how people represent regular and irregular for the past 40 years. for the single-route approach, @xcite described a feed-forward connectionist neural model that learned both regular and irregular forms of the english verbs' past tense without explicit symbolic rules. however, this model received fierce criticisms from the proponents of the dual-route model (e.g. @xcite , who argue that the speakers first reason over the abstract categories (regular -irregular), and process the regulars through rule-applying mechanism (adding -ed) and process the irregulars via gradient analogical processes. in addition, @xcite highlighted many empirical inadequacies of the model and argued that these failures stemmed from 'central features of connectionist ideology' and would persist in any neural network model. with the advancement of deep learning in nlp, there has been renewed interest in the english past tense debate with modern neural networks. @xcite revisited the past tense debate and showed that modern recurrent encoder-decoder (rnn) neural models overcame many of the criticisms. their model achieved near-perfect accuracy on the unseen regular verbs and some accuracy on the unseen irregular verbs (28.6% as 5 correct irregular verbs). in addition, the model's results on the nonce verb inflections correlate with human experimental data (spearman's ρ = 0.48 for regulars and ρ = 0.45 for irregulars). thus they concluded that the neural model could be a cognitive model. however, other studies have shown that the modern neural network is still susceptible to the criticism raised by @xcite : the neural models lack symbolic rule learning ability and are vulnerable to the frequency distribution of the data, so they may learn to extend the most frequent pattern, instead of the regular pattern. @xcite closely examined the model's performance on the nonce verbs and found that the fit to the human data is weak, especially for the irregular verbs. similarly, @xcite used german plural to demonstrate that the rnns tend to overextend the most frequent plural class to nonce words and do not match the human speakers' data. @xcite found that in english and german plurals, transformers are also susceptible to the frequency distribution of the data as rnns. prior work has generally focused on the comparison between model's performance and human behavior on nonce verbs, and few have explored the neural model's behavior on english regular and irregular verbs. in our study, we closely examine the transformer's behavior on english past tense inflections corresponding to the training data's regularirregular type and token frequency distributions to explore whether the models learn and apply symbolic rules. we train a set of transformers with different frequency distributions and experiment with resampling the training data for each epoch ( §4). on our evaluation ( §5.1) of english verbs, the transformers achieved over 95% accuracy on unseen regulars and some accuracy on unseen irregulars (ranging from 0% -22%). we find that models exhibit different behaviors on the regulars and the irregulars, that the performance on regulars is more affected by the type frequency but not token frequency, and vice versa for the irregulars, suggesting that the models have some degree of abstract representation of verb regularity. we observe that the majority of the errors can be attributed to misclassification (e.g., treating an irregular as regular), with a smaller proportion of errors caused by applying the wrong inflection. for nonce verb evaluation ( §5.3), the models vary in correlations with human data. generally, the models correlate with human data better on regulars than irregulars, but the overall correlations are weak. in conclusion, we found that the transformer models display some degree of abstract representation of verb regularity, but do not fit human data well, thus can not be a good cognitive model. ## data the base dataset is the same one used in previous studies with english past tense, which includes 4,039 english verbs from the celex database @xcite . we converted the verbs to ipa symbols based on carnegie mellon university pronouncing dictionary using eng-to-ipa python package, foot_2 and checked each verb's past tense forms on merriam webster dictionary. foot_3 among these verbs, 3,857 are regular verbs; 150 are irregular verbs; and 32 verbs have both regular and irregular forms, e.g. knitknit or knitted. 5 we also created two labels for each verb: regularity and verb class. the regularity indicates whether the verb is regular or irregular. the verb class corresponds to the inflection of each verb, which includes three classes for regular verbs (/-d/, /-t/, /-id/) and seven classes for irregular verbs, including vowel change, vowel change +/-d/, vowel change +/-t/, ruckumlaut, weak, level and other @xcite . the examples for verbs of different regularities and verb class labels in the base dataset are shown in ## discussion and conclusion in this work, we demonstrate that the transformer models exhibit some abstract representation of regular and irregular verbs in past tense inflection generation. this abstract representation is largely affected by the type frequency of the input data. since the regulars have a higher type frequency, the abstract representation is more robust for regular verbs than the irregular verbs. in addition, as long as the model could correctly classify the regular verb, it rarely makes errors in applying the correct inflection. given the low type frequency and highly diverse inflection patterns for the irregular verbs, it is challenging for the model not only to classify the irregulars correctly, but also to apply the appropriate inflections. we found that increasing the type ratio would improve classification, and increasing token frequency would improve applying the correct inflections. in addition, we also compared the model's nonce verb output with human data. the correlation with human data varies greatly for different models, which makes it difficult to state whether the neural models can capture human behavior. in our bestperforming model, we observe that the model is able to produce both regular and irregular forms for a nonce verb. however, the models are more influenced by their own regular or irregular bias than human speakers. for example, humans can generate various forms even with a strong preference for regulars. however, the models are likely to generate either regular or irregular forms for a certain verb. thus we conclude that the model's performance does not fit human data well. neural models have long been viewed as an approach against abstract representations. therefore, the neural models are often rejected as cognitive models. in our work, we showed that the models exhibit some abstract representations, although still have a weak correlation with human performance for different reasons. we hope our findings could imply that the dual-route mechanism is not necessarily against each other and lead to more discussions about incorporating both sides of the we created a dataset with parents' input past verbs with a higher irregular rate. we selected 8 children's corpora in the childes database @xcite and aggregated their parents' past tense verbs. these 8 children include adam, eve, sarah, peter @xcite , allison @xcite , naomi @xcite , and fraser @xcite . all 8 children have been extensively studied in the previous literature to show that they have overregularization errors at an early age. however, we didn't use it as one of our training sets, because this dataset is too small for training from scratch, including only 411 unique past tense verbs with 69 unique irregulars (irregular verb ratio is 16.8%). if we leverage the percentage of its irregulars with the same construction method of token irr , the dataset size would be 13,854 with an irregular ratio of 72.6%, which we used for the irregular ratio for reg ds .