Limits for learning with language models (2023)

Abstract: With the advent of large language models (LLMs), the trend in NLP has been to train LLMs on vast amounts of data to solve diverse language understanding and generation tasks. The list of LLM successes is long and varied. Nevertheless, several recent papers provide empirical evidence that LLMs fail to capture important aspects of linguistic meaning. Focusing on universal quantification, we provide a theoretical foundation for these empirical findings by proving that LLMs cannot learn certain fundamental semantic properties including semantic entailment and consistency as they are defined in formal semantics. More generally, we show that LLMs are unable to learn concepts beyond the first level of the Borel Hierarchy, which imposes severe limits on the ability of LMs, both large and small, to capture many aspects of linguistic meaning. This means that LLMs will operate without formal guarantees on tasks that require entailments and deep linguistic understanding.

## introduction the success of large language models (llms) has led researchers in nlp to harness llms trained on vast amounts of data to solve a variety of language understanding and generation tasks, and some have claimed that llms can solve any task that can be specified via prompting @xcite . while the list of llm successes is long, there have been several recent papers that provide empirical evidence that llms at least sometimes fail to capture important aspects of linguistic meaning @xcite @xcite @xcite . those who have dabbled in "bertology" with respect to linguistic meaning often have the feeling that fixing one llm deficiency just leads to the discovery of new ones. this paper provides a theoretical explanation of certain of these observed failings of llms. in particular, we prove that llms cannot learn the notions of semantic entailment or consistency as defined in formal semantics @xcite because they are incapable of mastering universal quantification. our work builds on @xcite ; @xcite ; @xcite , concerning the expressive power of neural networks, but we focus on the learnability of semantic concepts and use novel tools. our argument has widespread implications: not only does a general capacity to recognize semantic entailment and consistency underlie everyday conversational interactions, but the meanings of a great many common linguistic expressions depend on universal quantification. this set includes-but is certainly not limited to-a long list of quantifiers (every, some, many, most,... every other, ...), temporal adverbs (always, never, eventually) that are essential to planning @xcite , modal operators (possibly, necessarily,...), and certain discourse connectives and adverbs (therefore, if / then, except, because, ...). we begin in section 2 by contextualizing our claims in terms of expectations about the linguistic capacities and applications of llms. in section 3, we introduce the framework of continuation semantics, which will allow us to adapt certain notions central to truth-conditional semantics to the case of llms. section 4 lays out the core of our theoretical argument, focusing first on what is needed to learn universal quantification and then generalizing our argument to a wide range of linguistic expressions. our theoretical argument suggests that we should expect certain empirical failures from llms, and in section 5, we provide evidence that our predictions are borne out. section 6 concludes. ## context our results are particularly relevant to downstream tasks that require an agent to not only create fluent, creative and contextually relevant speech but also to act precisely based on the meaning of linguistic expressions and reliably recognize semantic inconsistency. for a robot that has been instructed (via conversation) to tighten every screw of a door, to never walk on an airplane wing, or to stop drilling immediately if certain conditions hold, acting appropriately requires being able to infer what do to based on the linguistic meaning of the words every, never, stop, immediately and if -and in these cases, getting things mostly right won't do, especially if lives or substantial economic loss are at risk. an important corollary of our argument is that while it might be tempting to separate reasoning and linguistic competence @xcite , the former is in fact inextricably tied to our ability to draw inferences based on linguistic content-not just on, say, mathematical or real-world facts. this in turn suggests that approaches which attempt to patch up knowledge deficiencies for llms by giving them access to external models @xcite will fall short in developing reliable models of linguistic understanding because llms fail to grasp the notions that underlie the very way that sentences (and actions) are woven together in conversation. empirical studies like @xcite show that llm failures to respect semantic entailment in question answering tasks follow from fundamental features of llm training; thus while extensive training and large data sets may improve llm results, performance will inevitably remain unstable and we should continue to expect hallucinations and reasoning errors in nlp tasks like question-answering and natural language inference. ## empirical investigations of llms with every while the theoretical argument laid out in section 4 does not hinge on empirical statistics of llm failures, it certainly suggests that we should expect such failures. in this section, we describe some of the tests we have performed using continuations to query llms directly about their mastery of universal quantification. let us return to our simple example from above, repeated here as (2): we used finite sequences of formulas as a context, e.g., a 1 is blue, a 2 is red , a 3 is red ,..., a i is blue to determine a model. we then asked an llm m whether (2) in this model, allowing us to gauge its behavior with respect to finite domains. bert-large and roberta-large already failed to reliably distinguish very small models (containing 2 and 5 objects respectively) in which (2) is true from those in which it is not. to test these models, we fine-tuned bert-large and robertalarge on the coqa dataset @xcite . for finetuning, the model had 4 output heads for yes, no, unknown, and span type questions. since the coqa dataset provides a rationale for each question, the models were jointly trained on question answering and rationale tagging tasks to enhance their performance. we report scores on the finetuned models on coqa for 1 epoch as we did not observe significant improvement with an increased number of epochs. for bert-large, we provided strings like (3) and then asked is everything blue? we generated a total of 5 examples in which (2) was true and 5 examples in which (2) is false. all the examples had only 2 objects. the inconsistent examples were constructed by varying the position of the object which was inconsistent with the asked question and by trying out different combination of colours and objects. the consistent examples were of the form: 1. the car is blue. bert-large was able to correctly identify the consistent examples but failed for all the inconsistent examples. as the model failed for all the inconsistent examples with 2 objects, we did not experiment with models containing more than 2 objects. for roberta-large, we generated a total of 9 consistent examples and 53 inconsistent examples. we constructed sequences ranging from 2 to 10 objects. for each number, the inconsistent examples were constructed by varying the position of the object in the string (context) which is responsible for the inconsistency. the model was able to correctly identify all the consistent examples. for models of a given size (i.e., number of objects), we defined the pass fraction as the ratio of the examples in which the model was able to report models inconsistent with (2) correctly to the total number of inconsistent examples. our empirical observations on llms like bert and roberta and probing of chatgpt strongly support our argument that llms are unable to master quantification, complementing observed llm difficulties with negation @xcite @xcite and to some extent quantifiers @xcite . ## conclusions we have shown that llms' demonstrably inadequate grasp of the meanings of words like every and other linguistic constructions has a theoretical foundation and explanation: for certain expressions s, s's content should be defined via consistent sets of strings in v ω , and llms cannot effectively learn certain sets in v ω . more generally, llms cannot effectively learn full meanings of first order quantifiers or any borel sets beyond the basic open sets, which means that they fail to grasp the meaning of a long list of mundane, frequently used expressions. many of these expressions are syncategorematic terms and express what we might call precise concepts. such concepts are needed for understanding ordinary entailment across all expressions; in addition, correctly reasoning with these concepts and grasping their entailments is essential to understanding them. reasoning and entailment are intimately tied with meanings. for us and most formal semanticists @xcite , grasping meaning and correctly reasoning with linguistically expressed concepts go hand in hand; if you cannot exploit the meanings of words in correct reasoning, you do not really know what they mean. the incorrect reasoning of llms exemplifies their failure to grasp semantic entailments and meaning. our arguments go beyond those of bender and koller (2020), who argue that stochastic models cannot capture linguistic meaning because they consider only form, not denotation. while we agree that denotation plays a very important role in meaning for many expressions, the meaning of most expressions, and especially that of syncategoreg-matic ones, requires us to capture their semantic entailments. we have shown that we can capture these entailments within the semantic framework of llms using continuation semantics. but we have also shown that llms nevertheless fail in this task. llms can learn certain types of ∆ 0 1 sets and finite intersections and unions of learnable ∆ 0 1 sets. for many open class words-including many nouns, adjectives and verbs-whose characteristic denotations can be determined given a finite sample, this probably suffices to capture their meaning or at least a very good approximation of it. in addition, many nlp tasks may not involve logical inference but an independent form of string optimization; in text summarization or translation, where given a context s, m tries to find an optimal continuation s ′ . if the length of s.s ′ falls within the constraints of corollary 4, then we can expect an llm to succeed at such a task. proposition 6 and corollary 4 generalize corollary 1 and they all point to a general limit on learnability for llms. they establish that language models have strict bounds even on the ∆ 0 1 sets they can effectively learn. so we cannot count on llms having full linguistic competence even on finite domains. different models may have different limits; smaller models generally with lower limits. this motivates a comparative study of the limits of learnability for different llms, complementing @xcite . because we do not make assumptions about memory but only about inductive processes and learning, our results hold for arbitrarily large llms and for any task that relies on an llm's capacity of string prediction, even if strings are not directly predicted. our research implies that full language mastery needs a different approach from one in which one seeks to build ever larger llms with language masking or autoregressive training. following @xcite , we believe we need to inject knowledge about linguistic structure and content into our models to further constrain learning and in particular hypothesis sets and learning architectures as suggested in steinert-threlkeld and szymanik (2019).