Language Models Understand Us, Poorly (2022)

Abstract: Some claim language models understand us. Others won’t hear it. To clarify, I investigate three views of human language understanding: as-mapping, as-reliability and as-representation. I argue that while behavioral reliability is necessary for understanding, internal representations are sufficient; they climb the right hill. I review state-of-the-art language and multi-modal models: they are pragmatically challenged by under-specification of form. I question the Scaling Paradigm: limits on resources may prohibit scaled-up models from approaching understanding. Last, I describe how as-representation advances a science of understanding. We need work which probes model internals, adds more of human language, and measures what models can learn.

## introduction a theme of emnlp this year is "unresolved issues in nlp." hence i consider what it means to understand human language, whether current language models understand and whether future models will. recent large language models have achieved impressive results on benchmark tasks @xcite . these results challenge ordained wisdom on the representations necessary for language production. we've seen improved results from multi-modal models @xcite @xcite @xcite , what some call foundation models @xcite . some models even run images, text, and games @xcite . @xcite identify language understanding and scaling as pertinent and much debated questions in nlp. so what's next? i identify three views on language understanding ( §2): understandingas-mapping, understanding-as-reliability, and understanding-as-representation. through examples of recent limitations of language models ( §4), i argue for understanding-as-representation because it climbs the right hill ( §3). in particular, i question the assumption that scaling current models is computationally feasible to lead to human-like understanding ( §5). because of the large gap between human and model understanding, i think it is generally misapplied to say that models "understand" ( §6.1). better applied are examples of promising work on understanding ( §6.2). ## views on understanding some argue that there is a strict barrier which separates human from machine understanding @xcite . understandingas-mapping puts understanding in terms of an absolute mapping between form and meaning. here, meaning comes from what a series of forms describes. those forms can be composed in a variety of ways to yield different, legible meanings. 1 often, those with this view imply humans have special access to meaning. others argue that we ought be rid of the distinction between human and machine understanding. they imply models will close the gap soon enough @xcite @xcite . understandingas-reliability puts understanding as a question of reliable communication: can one agent expect another agent to respond to stimuli in a certain way? foot_1 this view assumes that scaling alone will lead to an agent capable of human-like language; system internals don't matter. for example, in the most extreme case we can imagine a very large look-up table with state (cf. russell and norvig 2021): a mapping from every input sequence to a sensible output sequence. in this paper, i put understanding in terms of internal, dynamical representation: when prompted with a stimulus, does an agent reproduce an internal representation similar enough to that intended? call this understanding-asrepresentation. many have proposed related theories @xcite @xcite @xcite . in this view, if someone unthinkingly blurts out the correct answer to a question, they would not have understood. while a thermostat reproduces a certain representation given a temperature this representation is not similar to a person's. some have said that models appear not to understand because their interrogators fail to present stimuli in a model-understandable way @xcite . exactly: i am concerned with human language understanding-not any possible form of understanding. to advance a science of understanding, i argue that as-reliability is necessary, as-representation is sufficient, and as-mapping is neither. i reject the premise of as-mapping that the way we use words is separate from our meanings. while current work in nlp poorly approximates shared intentionality foot_2 i disagree that this is the only route to meaning. 4 we could imagine a very large lookup table. there is no boundary between what is and what is not a language. foot_4 i accept as-reliability in theory. enough data and parameters should yield a language-performant agent indistinguishably similar to a human tested on byte streams passed along a wire. similarly, @xcite argues that a self-supervised foundation model could do so. still, i am skeptical of what i call the scaling paradigm, that scale alone is a realistic approach. i think that hill climbing works but we're climbing the wrong hill. ## climbing the right hill as-representation and as-reliability are compatible: we may care about representation but more easily look for reliability. i argue that input-output behavioral tests are necessary but may not be sufficient to attribute understanding-we may need to look inside. 6 nonetheless, alisha, when messaging with bowen, has no need to look inside bowen's head to verify that he understood the following exchange: a: i'm unhappy. b: why aren't you happy? our human bias is to assume that other agents understand until evidence proves otherwise @xcite . this is pragmatic; until recently humans did not encounter non-human agents who could respond somewhat reliably. humans assume a similarity of representation, that others have the same inductive biases. we can't make that assumption with our models. we can't assume that a chat-bot has a bias to coo over babies (cf. hrdy 2009). this is why @xcite test doesn't work-the smoke and mirror programs which won the loebner prize unintentionally parody input-output tests @xcite . reliability, while useful, alone does not advance a science of understanding. as-reliability does not tell us which biases induce understanding. it is not causal. granted, humans' internal representations are difficult to measure, may change at each point of access, and in ai we've historically leaned too heavily on certain putative representations. @xcite calls this a "bitter lesson." so why talk of representation? i agree with the "bitter lesson" but i also know that there is no such thing as free lunch; human language occupies a small manifold in the space of possible functions. i don't argue to replicate natural functions but rather to be honest about human strengths lest we wander off into fruitless regions of state space. to do logic, at some internal level a system is going to have to appear to use the parts of logic. advancing as-representation does not mean we know what representations underlie human language nor that we must use certain ones. advancing as-representation does mean that we pay attention to the constraints on human language usage ( §4). we should use those to guide our benchmark tests for reliability. we should not get lost in our proxies, especially what the scaling paradigm assumes ( §5). ## under-specification of meaning language is dynamic (e.g. has a history), intersubjective (multi-agent), grounded in a large number of modalities (senses), collectively intentional (in a cultural context), and more. present models have little, if any, data on these aspects of language. as bisk et al. (2020) make clear in their "world scopes," the majority of work in nlp attempts to learn language from internet text alone. i agree that models have fewer data of the world than humans @xcite . what our models see under-specify our meanings. recent work has looked into such dissimilarities. @xcite note that while language models use novel constructions, they copy from training data at a high rate. mccoy et al. (2019) and @xcite identify how models use heuristics and short-cuts contrary to the human meaning of prompts. @xcite show that a current model will not remember a game after a long-enough context window. work has just begun to show limitations of multimodal models. @xcite test for compositionality over images and text: current models perform at chance. @xcite and conwell and ullman (2022) show many similar compositional issues for dall-e 2 in comparison to humans. lake and murphy (2021) note the implausibility of present multi-modal models: e.g. they cannot describe internal desires or change beliefs. thus my claim is not that models can't learn meaning. my claim is that for models to approach human meaning they will require data on aspects of language the field has only begun to investigate. ## challenges to the scaling paradigm but what of more data? the scaling paradigm tells us that scale alone-more parameters, more data, more modalities-will be enough to approximate human language understanding. exponential increases in parameters or training data have yielded prompt: are each of these events temporally earlier (a), present (b), or later (c)? still, text doesn't seem like enough. @xcite argue that there aren't enough examples to learn meaning from form in languages of assertions. even @xcite admit to be running out of clean data for exponentiallybigger models. furthermore, by the age of five, the average american child has heard between ten and fifty million words @xcite . a state-ofthe-art model sees from 10k to 100k more words than a kid. 7 for some (e.g. linzen 2020), the argument stops there: our language models must not be learning the correct functions because they require more data to generalize. nonetheless, humans have plenty of data to compare language use besides the words they hear @xcite . for example, @xcite show that children at first need items to be visually centered to learn them. so the claim that models learn the wrong function may only apply when limited to text. what of more modalities, then? to extrapolate on the figures of palm @xcite , we only need to scale a model to about 2 12 billion parameters and train it for 2.59 × 10 26 flops (and increase training data) for perfect performance on a variety of english nlp tasks. for perspective, @xcite estimates the cost of palm at 10 million usd which, at the 100x projected, is 1 billion. these figures seem infeasible. furthermore, the tests in palm do not capture the limits i mentioned for human-like understanding ( §4); we would need to add image @xcite and game @xcite networks as well. i don't know exactly what effect these added modalities will have except that they will increase the exponent on scale. i am wary that the long tail of returns embraced by the scaling paradigm chases an exponential. 8 the primacy of scale implies an exponentially diminishing longtail of capability. how soon until our models become planetary in size, as @xcite portends? i thus recast the debate, not as what can theoretically be learned from data (as the scaling paradigm trumpets), but as the computational efficiency of different learning approaches. to asymptotically approach an approximation of human language understanding, how many parameters, data, and modalities will our models need? these concerns about scale make me hesitant to suggest that efforts will soon close the gap between human and machine understanding (contra as-reliability) even as i agree that they will narrow it (contra as-mapping). ## conclusion present language models have limited access to meaning ( §4) and scale alone may not be sufficient to achieve human-level understanding ( §5), at least until we can guarantee similar representations or inductive biases ( §2). inference tasks reduce the space of possible entities implied by conventional usage. still, we must be assured of what they represent in order to guarantee their reliability. understanding-as-mapping would deny the distributional compositionality of our languages and of our minds. understanding-asreliability would claim our machines "get" meaning so long as we focus only on temporally limited usage. understanding-as-representation would focus on accurate, measurable, and tractable meaning.