Analyzing the Linguistic Priors of Language Models with Synthetic Languages (2025)

Abstract: While modern language model architectures are often assumed to be language-agnostic, there is limited evidence as to whether these models actually process the wide diversity of natural languages equally well. We investigate this question by analyzing how well LMs learn carefully constructed artificial languages containing a variety of verbal complexity, ranging from simple paradigms to covering far more verb classes than occur in natural languages. Rather than learning all languages equally efficiently, models trained on these languages show strict preferences for processing simpler languages. Furthermore, while some observed behaviors mimic human linguistic priors, we find that they indicate the model memorizes its training data rather than generalizes from it.

## introduction transformer-based language models (lms) are often assumed to be language-agnostic, or to learn all natural languages equally well. this has lead to their widespread use for different languages @xcite and multilingual modeling (e.g., @xcite . however, there is immense linguistic diversity in the world's languages, and human learners acquire aspects of these languages at different rates. for example, children take longer to learn the opaque dutch gender system, mastering it by age six @xcite , while children master the transparent spanish gender system by three and a half, if not sooner @xcite . it remains an open question as to whether this complexity similarly affects model acquisition of different languages: previous work exploring the differences in language modeling capabilities presents mixed results on the effect of morphological complexity on language modeling @xcite @xcite , and typological differences can impact the performance of models intended to be language-agnostic @xcite . furthermore, there is limited evidence whether lms are even constrained to learning natural linguistic phenomena as humans are @xcite . we address this question by testing if lms demonstrate human-like learning patterns when acquiring new, artificial languages. specifically, we ask: do lms exhibit linguistic priors favoring certain conjugation paradigms over others?. we center our behavioral analysis on a single grammatical feature-verb conjugation-in a wide variety of linguistically plausible and implausible settings as a controlled case study into the effect of linguistic grammatical complexity on transformer-based modeling of language. to evaluate lms for these linguistic priors, we first construct artificial languages using a probabilistic context-free grammar (pcfg). these languages cover a wide range of (plausible and implausible) conjugation complexity while controlling for other confounding variables found in natural languages. we then test how proficiently and efficiently language models learn these languages by measuring their mastery of both subject-verb agreement (a commonly used linguistic test for lms, see @xcite , as well as a novel behavioral experiment for verb class identification in these languages throughout the training process. our experiments find that language models acquire more complex languages (i.e., those with more verb classes) more slowly. however, they achieve close to 100% accuracy on seen verbs given enough data, even in cases where the number of verb classes is far larger than naturally occurs in human languages. the models also perform significantly worse on novel verbs than those seen during training, with the performance degradation increasing with the number of verb classes; this indicates that these models do not learn to generalize from the standard conjugation patterns shown to them during pretraining. these findings suggest both that (1) these models are not language-agnostic, but are instead sensitive to the complexity of the target language, and that (2) behavior that resembles human-like language learning in models may actually be memorization of the training data, rather than generalization to the underlying linguistic rules. put another way, correlations between model and human behavior do not necessarily indicate that their underlying mechanisms are the same. in light of these findings, we recommend future work analyzing model language learning to incorporate evaluations that disentangle these factors when probing language model behavior. ## methodology this section presents our method for generating artificial languages with the desired characteristics (section 2.1) and our behavioral experimental setting to test model proficiency on subject-verb agreement in these languages (section 2.2). ## results this section presents our language learning experiments. we find that while lms model simpler inflectional paradigms more easily (indicating that they are not agnostic to language complexity), they struggle to generalize to new verbs across experimental settings, including on linguistically plausible verbal paradigms found in natural languages. ## discussion this paper investigates whether lms exhibit linguistic priors for natural and unnatural conjugation paradigms. our probing experiments find that lms are much more efficient at modeling personnumber agreement for languages with simpler verb paradigms, mirroring human learning of languages. they also corroborate prior work indicating that neural lms prefer human languages to unnatural ones @xcite . however, this primarily holds for verbs seen during training; models perform much worse at judging subject-verb agreement on novel verb roots, in contrast with the strong generalization shown by human speakers on this task (e.g. @xcite . furthermore, we find that lms adopt unnatural verb paradigms foot_5 almost as well, given enough training data. this result, in conjunction with degraded performance exhibited on unseen verbs, indicates that model learning of the generated languages is likely heavily dependent on memorization rather than generalization of the training data, particularly in the class behavioral test setting. while this trade-off has been documented in lms for downstream nlp tasks @xcite , we find that it also affects the model when learning lower-level linguistic knowledge. even more unnaturally, models trained on languages with more complex paradigms are slightly better at identifying correct verb classes, with the best performance occurring on 32 and 64 classes. based on these results, we argue that while language model learning of verbal paradigms may resemble human learning, the underlying mechanisms driving these behaviors are likely very different. future work comparing model behavior with humans should control for these similarities by also looking at the underlying mechanisms driving model performance. ## limitations using carefully constructed artificial languages allows us to isolate syntactic complexity's effects on language learnability and to consider a broad, systematic complexity distribution. however, this means that these languages are not natural (particularly regarding the absence of semantics), which limits the findings presented here. future work should replicate these experiments in a more natural setting to verify that our findings remain valid in such conditions. another limitation of this work is the size of the language models: computational limitations and the number of models considered in our experiments (800 trained lms across experimental settings) limited the model size considered to one setting, gpt-2 small (124m parameters). finally, there are many aspects of complexity in natural language, with the number of verb classes being just one aspect. whether our findings hold for other linguistic phenomena, such as noun classes (i.e. gender), freedom in word order, degree of syncretism, morphophonological alternations, etc. remains an open area for future research.