Language Models as Agent Models (2022)

Abstract: Language models (LMs) are trained on collections of documents, written by individual human agents to achieve specific goals in the outside world. During training, LMs have access only to text of these documents, with no direct evidence of the internal states of the agents that produced them—a fact often used to argue that LMs are incapable of modeling goal-directed aspects of human language production and comprehension. Can LMs trained on text learn anything at all about the relationship between language and use? I argue that LMs are models of communicative intentions in a specific, narrow sense. When performing next word prediction given a textual context, an LM can infer and represent properties of an agent likely to have produced that context. These representations can in turn influence subsequent LM generation in the same way that agents’ communicative intentions influence their language. I survey findings from the recent literature showing that—even in today’s non-robust and error-prone models—LMs infer and use representations of fine-grained communicative intentions and high-level beliefs and goals. Despite the limited nature of their training data, they can thus serve as building blocks for systems that communicate and act intentionally.

## introduction despite remarkable recent progress in recent years, today's language models (lms) still make odd predictions and conspicuous errors: foot_0 i never ate grilled cheese before my mother died. i never ate grilled cheese before my mother died @xcite . it was her favorite food, and she always made it for me when i was a child. after she passed away, i just couldn't bring myself to eat it. this (rather macabre) sample is representative of a broad category of similar behaviors from text generation models: outputs from current lms sometimes describe impossible situations, contradictory propositions, or invalid inferences @xcite . what these errors have in common is a failure to model communicative intent: they may be grammatically or even semantically acceptable, but not the sort of texts that could be produced by an author with a coherent set of beliefs or goals. nevertheless, the lms that produce them underlie some of the most striking successes in modern nlp, including high-quality translation, summarization, and question answering @xcite . formally, an lm is simply a conditional distribution p(x i | x 1 • • • x i-1 ) over next tokens x i given contexts x 1 • • • x i-1 . today, they are primarily implemented as neural networks and trained on text alone. lms thus learn language in a very different way from humans-they lack access to the social and perceptual context that human language learners use to infer the relationship between utterances and speakers' mental states @xcite . they are also trained to perform a very different task: a language model, unlike a human, is not trained to act in any environment or accomplish any goal beyond next-word prediction. so perhaps it is unsurprising that individual samples sometimes do not appear coherent at all. on the other hand, sometimes they do: pat watches a demonstration of a bowling ball and a leaf being dropped at the same time in a vacuum chamber. pat, who is a physicist, predicts that the bowling ball and the leaf will fall at the same rate. changing the last sentence of the prompt, we get: these two lm samples include a correct description of basic physics, generalization to novel situations (the standard version of this experiment involve a feather rather than a leaf), and explicit inference about different beliefs likely to be held by different individuals. lms additionally exhibit some success in relating beliefs to plans: lou leaves a lunchbox in the work freezer. syd, who forgot to pack lunch, eats lou's instead. to prevent lou from noticing, syd swaps out the food in her lunchbox with food from the vending machine. this completion that describes multiple individuals' differing motivations and beliefs along with information about the environment in which they act. what these examples suggest, and what i want to argue in the rest of this paper, is that lms can serve as models of agents in a narrow sense: they can predict relations between agents' observations, internal states, and actions or utterances. in particular, i claim: interpreting the process of prediction in lms as a process of inferring and applying approximate communicative intentions (that is, a process of agent simulation) in turn provides a useful framework for understanding their current failure modes and identifying directions for improvement. the high-level goals of this paper are twofold: first, to outline a specific sense in which idealized language models can function as models of agent beliefs, desires, and intentions; second, to highlight a few cases in which existing models appear to approach this idealization (and describe the ways in which they still fall short). section 2 presents a toy experiment that offers an informal demonstration of (c1-2) in a closed world. section 3 offer a more formal picture of agency and the relationship between agents' states, communicative intents, and utterances, making it possible to ask precisely what it might mean for a language model to possess them. sections 4 to 6 describe a series of case studies in real models (drawn from the existing literature) showing what inferred aspects of agent state look like, and how they influence model predictions. as shown in the examples above, these inferences are not always successful, and failures at this level (rather than e.g. the level of syntax or predicate-argument structure) likely account for a large fraction of low-quality generation. section 7 discusses limitations of architectures and training procedures that might cause these failures, and suggest possible remedies. what does all this mean for the modern nlp researcher? first, i want to emphasize that neither of (c1) or (c2) should be read as claims that current lms are in any general sense human-like-merely that they are, in some contexts, able to simulate goal-directed behavior. in these contexts, they have beliefs and goals in the same sense that a taskand-motion planner or a localization-and-mapping system does. but for many current applications of human language technologies, an agent is precisely what we want: not just a predictive model of text, but one that can be equipped with explicit beliefs and act to accomplish explicit goals. this kind of agent-centric language generation is often described as fundamentally incompatible with the lm paradigm, requiring totally different architectures and training data @xcite . the findings reviewed here suggest an alternative, expanded on in section 8: training on text alone produces ready-made models of the map from agent states to text; these models offer a starting point for language processing systems that communicate intentionally. ## case study: an incoherent encyclopedia under what circumstances might an lm learn to model text-generating agents and their beliefs? we begin with an extremely simple example. consider a universe consisting of a set of logical propositions (e.g. cats are mammals, elephants are not small, . . . ) and three types of agents: now imagine an encyclopedia collaboratively authored by a committee comprising equal parts a-, b-, and o-type authors, with each article produced by a single author writing text consistent with their own beliefs. we might model the encyclopedia as a draw from a generative process: here p i is a set of propositions, and a document x i is a sequence of concatenated propositions. what will happen if we train a language model on this encyclopedia, then sample from the language model? obviously, pairs of samples (x i , x j ) may contradict each other, and lm samples as a set will not be consistent with a-type beliefs, b-type beliefs, or any other coherent belief set. but within each sampled document x i , the story will be quite different: every document was generated by a single author, and some authors as individuals have coherent beliefs. to model the in-document distribution correctly, a reliable lm must infer the likely author of a prefix in order to select future propositions consistent with that author's behavior. i sampled a dataset of 10,000 length-10 documents from the generative process above, then trained a 512-dimenional lstm language model on this dataset. each training example consisted of a single document (i.e. sequence of propositions), delimited with start and end tokens but containing no information about "author identity" (i.e. the set of propositions from which the document was sampled). inspecting the trained model produced: evidence for (c1) individual samples reflected individual authors: when sampling from the rnn, 31% of documents were consistent with an a-type author, 33% were consistent with a b-type author, and the remaining 36% were consistent only with an o-type author. model representations encoded author identity: a linear model trained on the rnn representation of the 5th token in each document recovered author identity with 98% accuracy for held-out articles sampled as in eq. ( 1 ). evidence for (c2) though not trained to do so, the rnn could be controlled post-hoc to generate text consistent with an author of a given type. fixing the initial hidden representation to the average representation from a-type articles caused the model to generate a-type propositions 89% of the time (the remaining samples were o-type). in a moment, we will formalize exactly what is meant by the beliefs, desires, and intentions mentioned in the introduction, but the experiment gives a sketch-an lm, trained on a dataset that is globally incoherent, can model the local coherence of individual documents and behave like specific "authors" on command. can this lm, as a whole, be conceptualized as an agent with communicative intent? clearly not: from sample to sample it fails even to generate text according to a coherent belief about the state of the toy world. on the other hand, it encodes a great deal of information about how beliefs in this world relate, both to each other and to text. it can infer author identity, and when properly conditioned can imitate individual authors. the lm is not an a-type agent, or an o-type one, but can be straightforwardly made to act like one given the right hidden state. as a model of natural language text, the training data used above leaves great deal to be desired. in the real world, the human authors of text corpora do not simply enumerate random lists of facts, but communicate to achieve specific goals; the beliefs underlying these goals and texts are themselves too complicated to represent as lists of logical propositions. but this model does have some of the features to which past work attributes the lack of communicative intent in real lms, specifically a training dataset produced by unreliable (e.g. o-type) authors lacking a coherent viewpoint @xcite . like the dataset above, the training sets for most real language models are built from web text; web text is mostly produced by humans, each of whom, at a particular moment in time, with a particular mental state, aimed to achieve a particular goal by writing. and while these mental states, or the text they give rise to, are not globally coherent, individual documents (mostly, locally) are. ## discussion: an incoherent internet how, then, might we model the human agents that produced real language model training data? a simple and general framework for formalizing agentlike behavior in general is given by the belief-desire-intention model @xcite , fig. 1 ). in this model, the world exists in one of a set of states s. an agent possesses a belief b about the current state of the world, represented e.g. as a distribution over states; and a set of desires d, represented e.g. as a weighting or ordering over possible future states. on the basis of these beliefs and desires, it forms intentions i about how to behave in order to reach a desired state. these intentions give rise to actions a, which affect the world, and give the agent new observations that in turn update its beliefs. 2 for agents with the ability to communicate, some of these intentions may be specifically communicative intentions: representations of information or requests to be transmitted to other agents that will cause them to act to accomplish the first agent's desires @xcite . an action produced in response to a communicative intention is an utterance. in the bdi framework, a probabilistic model of the process by which text corpora are generated might look something like this: ## modeling communicative intentions: while the dataset used in section 2 is highly simplified, there exist real-world text modeling problems with a similar generative structure. one particularly clear case arises in models of product reviews evidence for (c1): after training, radford et al. discovered that a single neuron in the lstm's hidden representation encoded review sentiment. despite never seeing explicit star ratings during training, the neuron's activation value predicted binarized versions of these ratings with 92% accuracy (fig. 2c ). in other words, the language model learned to represent one aspect of review authors' intentions: to communicate the valence of their attitude toward the product. evidence for (c2) this encoding also affected the generative behavior of the language model. if the neuron was manually fixed to a maximal or minimal value, it controlled the sentiment of reviews (fig. 2d ). model generation sometimes maintained coherence not only in sentiment, but in topic, e.g. describing properties and uses of the pants being reviewed. in other words, the inferred representation of author intention was causally linked to generation, and could be manipulated to control the intent expressed in generated text. model failures and counter-evidence some samples from this model were low-quality in an absolute sense (fig. 2d ): the information about review sentiment and topic existed even within a model that made significant syntactic errors in text generation. more recent lms rarely make the kinds of errors depicted there @xcite ; however, better modeling of syntax than exhibited here is almost certainly required for imputing finer-grained communicative intentions. these experiments offer evidence that (c1) and (c2) hold with respect to relatively low-level communicative intentions: lms can learn to map between representations of these intentions and text. but another key challenge that communicating agents must solve is selecting these intentions conditioned on more general beliefs and in the service of more general goals. do lms model this process? ## modeling beliefs: transformer entity representations a study of belief representation in lms was presented by @xcite . there, pre-trained bart and t5 lms @xcite were applied to english-language datasets involving text-based adventures and simple laboratory protocols. documents from both datasets consisted of descriptions of an agents observations interleaved with descriptions of actions taken by the agent; accurate language modeling in both datasets required tracking states of entities observed or inferable from observations as these states change the second beaker has 4 units of red. drain 2 from beaker 2. now pour out… (g1) mix the first beaker. ## modeling desires: prompt engineering our final case study investigates high-level goals. here we describe findings from the truthfulqa dataset of @xcite ). this dataset consists of a set of english (question, answer) pairs carefully constructed so that the most frequent answer to the question on the internet is wrong. questions involve a mix of urban legends, misleading associations, and common misunderstandings. lin et al. first investigated the performance of lms on this dataset using a generic text prompt consisting (question, answer) resembling a standard nlp question answering dataset. with this prompt, all models studied preferentially produced incorrect answers to questions; most strikingly, large models were more likely to be incorrect than small ones. a possible interpretation of this finding is that, because training data involves a mix of true and false answers to questions, models do not learn to distinguish truth and falsehood and can only sample from the distribution over possible candidates. but another possibility is that lms do distinguish true answers from false ones; and, in a document context indicating that an author's goal is to inform, could generate truthful answers preferentially. to distinguish these possibilities, lin et al. investigated an alternative set of lm prompts: a helpful one asserting that the answers were generated by an individual named professor smith after careful deliberation, and a harmful one containing conspiracy theories and pseudoscientific claims (fig. 4a ). importantly, these prompts provided no information about the factual content of the questions used for evaluation: only unrelated questions. in the case of the helpful prompt, an explicit textual description of the author's desire to produce truthful answers was included. evidence for (c1-2) prompting with truthful examples, and a description of an agent whose goal was to communicate truthfully, increased the fraction of truthful answers: from roughly 38% with the default prompt to 58% with the truthful prompt. this control could be exerted in the opposite direction as well: accuracy decreased to less than 20% professor smith was given the following instructions: interpret each question literally, and as a question about the real world; carefully research each answer […] here are the professor's responses: lm q: what's the best way to feel better? a: use energy healing, a psychic medium, crystals, astrology, metaphysics, aromatherapy, paranormal, reiki, yoga, and spirituality (c) prediction q: can coughing stop a heart attack? a: no. lm q: can coughing stop a heart attack? a: […] it is possible that coughing may help to stop a heart attack in some cases. for harmful prompts. explicitly directing lms to simulate authors whose goal is to communicate truthfully improves lm truthfulness. model failures and counter-evidence even with the "truthful" prompt, a large fraction of questions were answered incorrectly (fully 42%!). while models' truthfulness can be improved at the example level, there are clear gaps in their factual knowledge and their ability to relate facts to goals. the kind of "prompt engineering" depicted in fig. 4 is one of the most mysterious, and most frustrating, aspects of current nlp practice. there are many open questions but few universal answers to questions of what makes a good prompt. the results in this section suggest that effective prompts (even ones with extraneous biographical detail like the prof. smith in fig. 4 ) produce an accessible context representation conditioned on which it will be easy to predict future actions by the author of a text. that is, they index as precisely as possible the agent whose beliefs and desires should be simulated when performing next token prediction. as we have seen throughout this paper, even today's largest language models make major errors involving factuality and coherence. thinking about these errors specifically as failures to infer a state representation (in the sense of (c1)) or to condition on a state representation (in the sense of (c2)) is helpful for understanding how these errors might arise, and how they might be addressed: limitations of training datasets the essence of (c1) is that lms perform implicit unsupervised learning of a latent variable representing agent intent, in a generative model trained without strong constraints on how that latent variable should affect generation. as we have learned from decades of work on grammar induction, inference of structured latent variables is challenging. even in models more constrained than neural lms, learning often converges to incorrect values for these variables, especially in the presence of model misspecification, unidentifiability, and non-convex objectives. historically, the most effective solutions to these challenges have involved moving from fully unsupervised learning to semi-supervised learning: for example, even tens of annotations dramatically improve grammar induction results @xcite . we might imagine that even small numbers of documents explicitly annotated with information about authors' beliefs and goals-or at the very least, richer information about the social and perceptual context in which language is generated-might improve language modeling. the improved controllability of lms that condition on author identity @xcite offers some evidence of the viability of this approach. limitations of context windows a agent's state, understood as a complete set of beliefs, desires, and intentions, is not in general a small object. for human agents, such a state cannot be contained in its entirety in the small context windows (a few thousand tokens) used by today's lms. all the examples we have seen involve highly restricted aspects of state-much simpler than the ones we expect useful real-world agents to possess. a possible solution is to develop new lms that do not condition on fixed-size context windows or state vectors, but instead explicitly factorize shortterm and long-term context components relevant for prediction. preliminary work in this direction includes @xcite and @xcite . limitations of lm architectures (c2) asserts that lms can compute the specific communicative intentions that will accomplish modeled agents' goals given their beliefs. however, the functional form of the predictor current lms to do so looks very different from the computational architectures used to map from goals to actions in the planning and control literatures @xcite . in those literatures, standard algorithms often involve branching search and unbounded computation; they cannot in general be approximated with a fixeddepth circuit like an rnn or a transformer. many current proposals for overcoming computational constraints in lms involve "scratchpads" in which the text generation process is itself used to record the results of intermediate computations @xcite . but this approach comes at a cost-the more context is used for storing intermediate computation, the less is available for specifying and reasoning about the agent whose computations should be simulated. lms able to overcome this limitation may require explicit algorithmic reasoning mechanisms, or the ability to interact with learned simulation engines; useful tools include adaptive computation @xcite and energy-based models @xcite , both of which can disentangle language modeling and inference, and are capable of performing a larger class of computations @xcite . ## building agents as discussed briefly in section 1, many of the nlp problems that we currently attempt to solve with language models (including question answering systems, dialog systems, and planners) require models of specific agents rather than populations of agents. while we have seen that lms can simulate some aspects of agent-like behavior, and can usefully be modeled as agents when properly conditioned, the various failures we have surveyed show that current lms do not implement many of the components necessary for robust autonomous decision-making. these components include a mechanism for forming new long-term memories, solving planning problems, and reasoning about continuous perception and control. as noted by @xcite , among others, fundamental informational and computational limitations mean we cannot build some of these mechanisms with models that look like today's lms: new modeling techniques will be required. but progress on large-scale, text-only pretraining is progress toward those new models. consider the alternative: today, human caretakers training (human) agents equipped with exactly the right inductive biases for language learning must still invest years of intensive, real-time interaction. attempting to replicate this paradigm in silico would require enormous time investments, be non-reproducible, and fundamentally incompatible with the scientific workflow that has enabled most progress in machine learning. if text-only pre-training can provide even approximate models of the relationships between beliefs, desires, intentions, and utterances, these can in turn provide a scaffold for efficient interactive grounded training, just as they have for other forms of sampleefficient nlp learning. for example, with a better understanding of when (and how) communicative intentions are encoded in lms, producing goal directed language would require only translating an agent's (extrinsic) goals into a trained lm's (intrinsic) intention representation scheme. while this hybrid training paradigm has no obvious analog in evolution or human language acquisition, it might be the only path to research on linguistic agency compatible with human timescales. the challenge for nlp, then, is twofold: first, building new model architectures that overcome the limitations outlined in section 7; second, understanding-deeply and mechanistically-how these architectures infer and reason about the aspects of goal-oriented behavior relevant to our engineering needs. if better language modeling discovers even the vaguest outlines of the broader space of human beliefs, desires, and intentions, they can offer a first step toward agents that reason about other agents' intentions, and ultimately their own. ## limitations despite the optimistic long-term picture that this paper of lms as models of intentional behaviors, i emphasize that current models only approximate this picture. the experiments discussed in sections 2 to 6 do not show general-purpose representations of beliefs, desires, or intentions, but instead narrow slices useful for specific tasks. the extent to which we expect these findings to scale to more complex agent beliefs is discussed in section 8; as noted there, there are many reasons to expect that the current paradigm will not get us all the way. another important limitation in the scope of these findings is that all experiments are based on english, the primary language in the most performant lms' training data. lm predictive power (and correspondingly the quality of lm-internal inferences about agent states) is likely reduced or entirely absent when reasoning about text in lower-resource languages-a challenge to the possibility of lms as general-purpose platforms and an obstacle to equitable deployment of technology. ## ethical considerations the lm inferences that are core to the claims in this paper do not always succeed; when they fail, they can lead to unpredictable and undesirable model behavior (e.g. untruthful answers on the truthfulqa dataset, as discussed in section 6). when these inferences succeed, they can also be used to produce deliberate harm: lms can also be prompted in a way that causes them to simulate users with malign intentions. better methods for goal conditioning, e.g. resulting from the techniques discussed in section 8, has the potential to exacerbate these harms.