Mission: Impossible Language Models (2024)

Abstract: Chomsky and others have very directly claimed that large language models (LLMs) are equally capable of learning languages that are possible and impossible for humans to learn. However, there is very little published experimental evidence to support such a claim. Here, we develop a set of synthetic impossible languages of differing complexity, each designed by systematically altering English data with unnatural word orders and grammar rules. These languages lie on an impossibility continuum: at one end are languages that are inherently impossible, such as random and irreversible shuffles of English words, and on the other, languages that may not be intuitively impossible but are often considered so in linguistics, particularly those with rules based on counting word positions. We report on a wide range of evaluations to assess the capacity of GPT-2 small models to learn these uncontroversially impossible languages, and crucially, we perform these assessments at various stages throughout training to compare the learning process for each language. Our core finding is that GPT-2 struggles to learn impossible languages when compared to English as a control, challenging the core claim. More importantly, we hope our approach opens up a productive line of inquiry in which different LLM architectures are tested on a variety of impossible languages in an effort to learn more about how LLMs can be used as tools for these cognitive and typological investigations.

## impossible languages core to our experiments are the set of impossible languages we synthesize. in constructing these artificial counterfactual languages, we consider their information-theoretic attributes relevant to machine learning, such as entropy rate, as well as their formal linguistic characteristics, such as adherence to hierarchical grammatical structures. we believe that our choice of languages broadly spans the impossibility continuum hypothesized in figure 1 . concretely, we specify impossible languages by defining perturbation functions of english sentences. these perturbation functions map english input sentences to sequences of tokens. we categorize our languages into three classes: *shuffle, *reverse, and *hop, defined in the next subsections. each class has one control language that represents unaltered english, or a pattern that is very similar to english. ## experiments we run several experiments to assess gpt-2's learning of our impossible languages. our first experiment (section 4.2) uses perplexities as a general evaluation to compare how well each impossible language model has learned its own perturbed language and see whether this reflects the hypothesized impossibility continuum. in our second and third experiments, we conduct a closer examination of the *hop languages. given that their count-based verb marking rules appear to be the least clearly implausible among our proposed languages, we focus on examining these rules specifically through targeted assessments using surprisal theory (section 4.3). finally, we dive deeper into the mechanisms each *hop model uses to predict their respective verb marking rules using causal abstraction analysis (section 4.4). for all evaluations, we run tests on several model checkpoints to observe the learning process over intervals of training steps. foot_3 ## discussion and conclusion contra claims by chomsky and others that llms cannot possibly inform our understanding of human language, we argue there is great value in treating llms as a comparative system for human language and in understanding what systems like llms can and cannot learn. prior explorations of neural language models have already been fruitful for understanding the generalization of syntactic principles from data @xcite @xcite @xcite . our paper complements this line of work. we have shown that gpt-2 models do not master our set of synthetic impossible languages as well as natural ones, challenging the unfounded assertions stated previously. even in the absence of a clear definition of what constitutes a possible or impossible language, we believe that our investigations advance this debate regarding llms. the lack of a definition does not hinder inquiry into this topic; in fact, it beckons further explorations of the boundary between the possible and impossible languages, as shown in our hypothesized continuum in figure 1 . we believe that the *hop languages we propose closely approach this boundary. at the same time, conclusions about llms' linguistic competence and preferences for natural languages should be informed by an understanding of the ways that models fundamentally differ from humans. for instance, we saw that models can perform operations that involve counting tokens because llms rely on tokens as basic units. while humans are sensitive to morpheme boundaries and word boundaries, it is unlikely humans rely on atomic tokens in the way that llms do. this does not mean that llms can fundamentally tell us nothing about human language. rather, as we did here, it is valuable to consider and control for this difference before making generalizations. since at least the 1950s, a major line of linguistic inquiry has focused on what aspects of syntactic structure can be learned just from data, without domain-specific innate priors (e.g. a universal grammar). llms lack strong in-built linguistic priors, yet they can learn complex syntactic structures. while many llms are trained with vastly more data than children see, there is increasing evidence that even systems trained on smaller amounts of data can learn interesting linguistic information @xcite . the current paper raises further questions along similar lines. since we do find that real languages are more learnable by gpt-2, this leads us to wonder what inductive bias of gpt language models matches natural language. we believe that this inductive bias is related to information locality, the tendency for statistical correlations in text to be short range. information locality arises in gpts due to their autoregressive training objective and has been argued to arise in humans due to the incremental nature of real-time language processing @xcite . since llms have been shown to learn the complex structures of human language and have a preference for learning such structures over unnatural counterfactuals, it follows that they are clearly relevant to investigations and claims about the necessary innate priors for language learning. arguments that they are "by design, unlimited in what they can 'learn"' and "incapable of distinguishing the possible from the impossible" @xcite do not offer convincing evidence otherwise.