Probing Neural Network Generalization using Default Patterns (2025)

Abstract: Whether neural-net models can learn minoritydefault patterns has been a matter of some controversy. Results based on modeling real human language data are hard to interpret due to complexity. Therefore, we examine the learning of a simple artificial language pattern involving defaults using three computational models”:" an Encoder-Decoder RNN, a Transformer Encoder, and a Logistic Regression. Overall, we find that the models have the hardest time with minority defaults, but can eventually learn them and apply them to novel words (although not always extend them to completely novel segments or novel CV-sequences). Typefrequency has the largest effect on learning in all models, trumping the effect of distribution. We examine the weights of two models to provide further insights into how defaults are represented inside the models.

## introduction some linguistic patterns (e.g., english past tense regular inflection) are best stated using default rules -rules for a heterogeneous category that apply as a last resort when more specific rules are not applicable. an early connectionist model of morphological learning by @xcite was criticized by proponents of rule-based approaches for its inability to appropriately handle defaults @xcite . in particular, it was claimed that the default behavior of the model could only hold under the condition of high frequency of the default (regular) category (although see work by @xcite ). more recent work by @xcite ) using an encoder-decoder (ed) rnn model also focused on the english past tense problem. however, @xcite found that this model was unstable across simulations and the averaged aggregated performance still did not provide a good fit to human data. @xcite , using the same model as @xcite investigated the learning of the german plural allomorphy, which has been claimed to involve a minority default pattern @xcite . they found that while their model achieved 88.8% accuracy on the held-out dataset, it did not match human performance in a few critical ways. in particular, unlike humans, it failed to generalize the two apparent default suffixes -en and -s (where -s is the least frequent suffix) to unusual novel words. another study compared the behavior of an ed model and a transformer model on both the english past tense (majority default) and the german plural (minority default) @xcite . the fit to human data was still weak, and the models were susceptible to some analogical errors. the modeling work described above involving actual languages is hard to interpret since so many factors play a role: for example, it is controversial to what extent the german -s suffix is in fact a default @xcite , the specific patterns are often lexically idiosyncratic and the factors conditioning the suffixes range from semantics to phonology and not all of them are included in models' training. in this work, we aim to address these problems by focusing instead on artificial languages designed to isolate the specific patterns of interest. our goal is to abstract away from the irrelevant complexities of natural language and focus on understanding how default patterns (including minority defaults) are represented and learned in a neural net model. we define "default" to be a category whose distribution cannot be described with a simple conjunction of features, but can be stated either as (a) a complement of other conjunctive categories or (b) a complement of the distribution of a set of idiosyncratic lexical items (exceptions). note that this is a definition of what a default distribution looks like in the data, not how it is represented in the mind/model. in this paper, we focus on testing case (a) above. our research questions are: what we find is that all of the models we tested are able to learn and represent default patterns, including the minority default. however, minority defaults are learned slower compared to any other category. for all of our models frequency has the largest effect on learning: low-frequency categories are learned slower compared to high-frequency ones. however, the effect of the distribution also plays a role, albeit a smaller one: patterns that have a wide heterogeneous distribution (defaults) are learned slower compared to those with a more narrow distribution. these results are in line with human learning as we will discuss later. additionally, the models made some errors on novel stimuli that included segments not seen in the training or cv-combinations not seen in the training. the rulebased models predict that true defaults should occur in all unusual circumstances, or when the learner does not know what to do. from the three models we tested, the transformer model was the closest to this type of behavior (but more testing is required). overall, we conclude that neural networks can represent minority defaults, but these patterns are difficult to learn both because they have low frequency and because they are more variable by having wider distribution. we then generated several versions of this pattern varying the frequency of the global default category. in the equal frequency condition, each suffix was equally frequent, in the majority default condition, 90% of words had suffix c, and in the minority default condition, suffix c occurred 10% of the time. ## the models we used three classification models, an encoder-decoder rnn, a transformer encoder and a multiclass logistic regression. for all of these models, the inputs was a sequence of numerical feature vectors representing sequences of sounds. for example, the sound [t] corresponds to a vector that includes a value of -1 for the feature corresponding to its voicing, a value of +1 for the feature consonantal, a 0 for the feature corresponding to its roundness (since that feature is only relevant to vowels), and so on (see appendix for the features we used). outputs used a single vector of three numerical features, each of which corresponded to one of the suffixes. an output value of 1 in training indicated that that feature's suffix was being assigned to the input stem. foot_1 ## results the accuracy at the end of learning for the three models (omitting lr-pc) is summarized in ## discussion it appears that all models we looked at were able to learn the simple artificial language pattern they were trained on, and successfully generalize to novel items that looked similar to training data. however, when tested on items that contained novel word-final segments, or novel cv-templates, the models produced some errors. in particular, all models struggled with the novel segment 'l' in the minority default condition, because this segment is expected to belong to the default pattern which was exemplified only by 10% of the data and because this segment is featurally similar to segments in the sibilant class (by being a coronal consonant). when the default class is more frequent, the models are able to overcome this difficulty and correctly categorize inputs ending in 'l'. the transformer model also had some difficulties with the novel segment 'h' in a condition in which the category to which it belongs (suffix b) is low frequency. another source of errors were words from different templates: the ed model struggled with the novel cvcc template, while the transformer model struggled with the vc template. we hypothesize that the humans would not have difficulties with such novel templates and would generalize based on the features of the last segment. however, the current models are either not abstracting away the concept "last", or are making their decisions based on the shape of seen words, and not just the features of the last segment. our analysis of the two models' weights revealed that different categories have different assignment of feature weights so as to maximize the probability of an input falling into a specific class if it shares more highly weighted features with that class compared to all the other classes. this inherent competition among classes is important for creating the apparent default structure. the networks find such weight configurations so as to ensure that inputs sharing features with both a default and a narrow class, will be more likely to be assigned to the narrow class (capturing the principle that the more specific classes "block" to the more general ones). this typically means that the cumulative weight of all relevant features in the default class will be lower than the cumulative weight of features in the non-default class. overall, we also see that the frequency of a pattern has the largest effect on learning: unsurprisingly the more frequent the pattern, the faster it is learned. when frequency is held constant, patterns that have a more narrow distribution in the phonological space are learned better. this is consistent with human data. for example, studies on category learning in the domain of visual perception report that more diverse categories with a wide distribution are learned worse than those with a narrow, clustered distribution @xcite . artificial language learning studies that investigated the effects of frequency and distribution on learning defaults find the same thing @xcite . the reliance of network models on both frequency and distribution has also been found before, including in work on modeling minority defaults. in particular, @xcite found that testing a simple three-layer perceptron trained on german plural with inputs that had all features removed produced responses that were either -n (the most frequent suffix) or -s (the most widely distributed suffix). we are pursuing several directions for extending this work. first, we are considering alternative ideas for testing the "across-the-board" property of defaults that are predicted by some rule-based models. in particular, it has been argued that regular inflection (of the default type) will automatically apply to abbreviations, acronyms, proper names, non-native sounds, quotations, and when memory or processing fail @xcite . it is not obvious what the best analog of such circumstances would be for our models. but one idea we are pursuing is testing our models on some words with an entirely new subset of features which were set to 0 in the training, but which are contrastive in the testing. in our current setup, the closest test for generalization to "unusual" cases are the novel templates and novel segments. as previously mentioned, the ed model produced seemingly random behavior when tested on cvcc templates, affected by the absence of word-final clusters in the training data. on the other hand, the transformer model struggled with the novel vc words and 'h'-final words in one of the conditions. interestingly, this model tended to use the default suffix when making errors (although not always), which is more similar to the default-behavior one would expect from a rule-based model. thus, we take this as preliminary evidence that global defaults (those that would apply to any new or unusual situation) can arise in a non-symbolic system. more testing (along the lines suggested above) would be required to confirm this claim. another extension of our work is to make the target pattern gradually more complex, and therefore more realistic. instead of a structure where the "exceptions" (the more narrow categories) are deterministically defined, we'd like to test a situation that is more similar to the english past tense, namely when the exceptions do not form a natural class and partially overlap with the default category (e.g., monosyllabic verbs ending in coronal stops are typically irregular like put, but can also be regular like bat). finally, in our experiments we only varied typefrequency, while the token-frequency of each word was the same (=1). in the future, we would like to also investigate how the models respond to changes in both type and token frequencies as these are known to play different roles in language acquisition.