Modeling Overregularization in Children with Small Language Models (2024)

Abstract: The imitation of the children’s language acquisition process has been explored to make language models (LMs) more efficient.In particular, errors caused by children’s regularization (so-called overregularization, e.g., using wroted for the past tense of write) have been widely studied to reveal the mechanisms of language acquisition. Existing research has analyzed regularization in language acquisition only by modeling word inflection directly, which is unnatural in light of human language acquisition. In this paper, we hypothesize that language models that imitate the errors children make during language acquisition have a learning process more similar to humans. To verify this hypothesis, we analyzed the learning curve and error preferences of verb inflections in small-scale LMs using acceptability judgments. We analyze the differences in results by model architecture, data, and tokenization. Our model shows child-like U-shaped learning curves clearly for certain verbs, but the preferences for types of overgeneralization did not fully match the observations in children.

## introduction current llms require a huge amount of data for training, and this is an issue in terms of data collection cost and training time. in contrast, infants can acquire their first language with low resources. one of the llms, gpt-3, requires approximately 200 billion words for learning @xcite , whereas infants are required to learn only about 100 million words @xcite . recent studies have shown the benefits of mimicking human language acquisition. for instance, using child-oriented vocabulary and/or child-directed speech (cds) as learning data improves learning efficiency (e.g., @xcite . in the language acquisition literature, there are many studies on children's errors and u-shaped learning curves (bybee and i eated apples yesterday. i ate apples yesterday. ## our approach in this study, we analyze the neural lms using cds as the training data and investigate what generalizations the model prefers at each step in the learning process. in previous studies, the model learned direct transformations of verbs from the base forms to the past forms, so it was sufficient to examine whether the generated past tense was correct. however, since this study does not directly learn the generation of past tense forms, an alternative evaluation method is necessary. therefore, we adopt relative acceptability judgments for minimal pairs @xcite , as illustrated in figure 1 , to evaluate the model's preferences by comparing the generation probabilities. such evaluation methods have become widely used in the context of lm probing @xcite . to assess children's error preferences, we create minimal pair data of children's errors, consisting of sentence pairs with and without overregularization. in this task, the model is forced to generate these pairs. if the probability of generating sentences with overregularization is higher, it indicates that overregularized forms are preferred over correct irregular forms; conversely, if it is lower, it indicates that overregularized forms are not preferred. we follow blimp @xcite to generate sentences with over-regularization, which results in an evaluation dataset consisting of 1,000 minimal sentence pairs for the linguistic phenomenon we are investigating. we focus on a typical phenomenon of overregularization in children: the inflection of english past tense. for irregular verbs, we first create pairs of overregularization forms (e.g., @xmath0. next, we create pairs of sentences containing the overregularization form and sentences containing the correct past tense. the overregularized forms are created by concatenating the bare form of the verb with -d or -ed (base+ed) and the past form of the verb plus -d or -ed (past+ed) in a rule-based manner. the list of overregularization forms created is shown in ## experiments we train a small-scale version of gpt, an lm with incremental learning, on cds using a characterlevel tokenizer and evaluate the model on our evaluation data of past-tense inflections in english verbs. ## discussion how well do lms reproduce children's u-shaped learning curve? as we saw in section 5, the learning curves of the models trained with cds show that, for certain verb types, there are u-shaped curves corresponding to the three stages observed in children. for other certain verbs, even if the models do not show u-shaped learning curves, the accuracy still matches children's observations. however, in many of the verb types, the model did not reproduce the errors preferred by the children. even the verb type iv, which shows the most child-like trend in the model, did not perfectly reproduce the child's learning curve. however, as shown in figure 6 , the performance varied depending on the verb. specifically, verbs with a small character-level edit distance to their past tense form exhibited high performance and clear u-shaped learning curves. based on these findings, our model generalizes in a human-like manner in the learning of some verbs. this observation of a u-shaped curve in sub-regular verbs is consistent with previous studies @xcite . additionally, @xcite reported that the trend of the u-shaped curve was not observed in all irregular verbs. therefore, the results of our study are consistent with their findings. detailed results by verb can be found in appendix a.5. these results indicate that the model's performance is not entirely influenced by the frequency of the verbs. when we measured the correlation between the performance for each verb and the verb frequency, it did not show a clear correlation. detailed correlations can be found in appendix a.5.1. we then discuss the oscillations of the learning curve. @xcite argued that when interpreting u-shaped developmental patterns, it is important to distinguish between macro and micro u-shaped curves to quantify the acquisition of inflectional systems such as the english past tense. they noted that many irregular forms oscillate between correct and overregularized forms with micro u-shaped curves. @xcite reported the results of learning verb past tense inflection with a recent neural model and claimed that the learning curve does not show a macro u-shape, but a micro u-shape. our results in section 5 also show oscillations. however, learning in a transformer is more unstable, especially with small models or training data, and our results also show that the learning curve always oscillates at any point in the learning. we therefore conclude that the oscillations indicated by our results do not necessarily reproduce the characteristics of children's errors. our results better replicate children's language acquisition compared to @xcite , not by showing oscillations in the learning curve, but by demonstrating a macro-level u-shaped learning curve. instability in the learning process makes it difficult to conduct reliable analysis. in the future, we plan to try to eliminate the instability to make a more reliable analysis. why cannot lms prefer the correct past tense? as we saw in section 5, lms did not per-fectly select the correct past tense, even at the end of the training phase; some verbs showed unstable learning curves, and some selected overgeneralized forms until the end. the training data included the correct past tense, and the overregularized form, i.e., the negative example, was a non-existent word that should not have been included in the training data. therefore, it should be easy to select the correct past tense at the end of the training, but lms could not. one possibility for the cause of this problem is that the model does not capture the meaning of the task. because not all the sentences in the evaluation data have words related to the past tense (e.g., yesterday), the model may not understand the task of assigning a higher score when the correct past tense is used. if this is the case, it is natural for the model to assign higher generation probabilities to sequences that are closer to the original verb forms that occur frequently in the training data. another possible cause is that the setting of model training may not be sufficient for the model to learn the correct past tense. our results suggest that the learning of small-scale models is generally unstable and that cds is not enough for models to learn the generalization of correct past tense. the cds data we used for training, ao-childes, was taken from dialogue data between children and their parents. in our results, wikipedia articles probably substituted for children's input other than cds, making learning slightly easier. the results of the subword-level model with divided inflectional forms in figure 2 support this possibility. however, while wikipedia articles provide additional information, it is not the same as the content of conversations between adults that would be input for children. more training data or more parameters may be necessary for the model to learn generalization. we would like to confirm this possibility in future experiments. note that the model of @xcite achieves nearly 100% accuracy on the training data, outperforming our model. however, it is important to note that their model learns to transduce verbs to past tense forms, which is a relatively simpler task. in contrast, our task involves training a language model on cds and evaluating it by requiring the model to assign higher probabilities to sentences with the correct past tense as acceptability judgments. this distinction highlights that their model addresses an easier task compared to ours. therefore, we cannot conclude that our model is inferior to theirs based on this comparison alone. how well do lms reproduce children's error types? lms generally preferred base+ed to past+ed when making errors, and past+ed was rare, both consistent with the observations of the children. however, in all settings, including the verb type iv (figure 4b ), which was observed to correspond with the three stages of children's language acquisition, lms showed no correspondence with the trend in the production of error types in children as shown by @xcite . our results suggest that even recent lms, which have shown efficiency improvements with cds, cannot reproduce the error type of the children's learning process. although we only considered two error types in our experiments, the previous studies (e.g., @xcite analyzed other error types such as copying. covering these other error types in the analysis may provide further insights. ## conclusion we carried out the analysis of verb overregularization in neural lms by acceptability judgment and reported the trend of overregularization in current neural lms. our model shows a macro ushaped curve corresponding to the three stages of children's language acquisition, not just oscillations. therefore, our model better replicates children's language acquisition than kirov and cotterell (2018)'s model. furthermore, the error types our model generally prefers match @xcite 's observations of children. however, our results differ from children's observations, which show a shift in error type preferences as learning progresses. additionally, our results do not match those of @xcite and @xcite , which show that models use nearly 100% correct past tense for many verbs by the end of learning. to solve these problems and conduct a more detailed analysis, we will use a larger model or training data more similar to a child's input. in addition, we plan to analyze the conditions required to replicate children's error type preferences. we also plan to conduct phonological-based experiments, a wug test @xcite to test the regularization performance of new words, and evaluations across multiple error phenomena. a.5 learning curve for each verb figures 10, 11, and 12 show the learning curves for each verb when the character level nanogpt (10.99m) is trained with ao-childes. each line in the graph shows the results of the trials with different seeds. for some verbs (lose, have, make, read, break, wake, and come), we observed a u-shaped learning curve similar to that of children. additionally, we observed high performance for these verbs towards the end of the learning process. these verbs share the characteristic of having a small character-level edit distance when converting to their past tense forms. these results are likely since our experiment used a character-level model rather than a phonological one.