Implicit Representations of Meaning in Neural Language Models (2021)

Abstract: Does the effectiveness of neural language models derive entirely from accurate modeling of surface word co-occurrence statistics, or do these models represent and reason about the world they describe? In BART and T5 transformer language models, we identify contextual word representations that function as *models of entities and situations* as they evolve throughout a discourse. These neural representations have functional similarities to linguistic models of dynamic semantics: they support a linear readout of each entity’s current properties and relations, and can be manipulated with predictable effects on language generation. Our results indicate that prediction in pretrained neural language models is supported, at least in part, by dynamic representations of meaning and implicit simulation of entity state, and that this behavior can be learned with only text as training data.

## introduction neural language models (nlms), which place probability distributions over sequences of words, produce contextual word and sentence embeddings that are useful for a variety of language processing tasks @xcite . this usefulness is partially explained by the fact that nlm representations encode lexical relations @xcite and syntactic structure @xcite . but the extent to which nlm training also induces representations of meaning remains a topic of ongoing debate @xcite . in this paper, we show that nlms represent meaning in a specific sense: in simple semantic domains, they build representations of situations and entities that encode logical descriptions of each entity's dynamic state. you see an open chest. the only thing in the chest is an old key. there is a locked wooden door leading east. next, you… (c1) …use the key to unlock the door. you pick up the key. consider the text in the left column of fig. 1 . sentences (a) describe the contents of a room; this situation can be formally characterized by the graph of entities, properties, and relations depicted in (a ). sentence (b), you pick up the key, causes the situation to change: a chest becomes empty, and a key becomes possessed by you rather than contained by the chest (b ). none of these changes are explicitly described by sentence (b). nevertheless, the set of sentences that can follow (a)-(b) to form a semantically coherent discourse is determined by this new situation. an acceptable next sentence might feature the person using the key (c 1 ) or performing an unrelated action (c 2 ). but a sentence in which the person takes an apple out of the chest (c 3 ) cannot follow (a)-(b), as the chest is now empty. formal models of situations (built, like (a )-(b ), from logical representations of entities and their attributes) are central to linguistic theories of meaning. nlms face the problem of learning to generate coherent text like (a-c) without access to any explicit supervision for the underlying world state (a )-(b ). indeed, recent work in nlp points to the lack of exposure of explicit representations of the world external to language as prima facie evidence that lms cannot represent meaning at all, and thus cannot in general output coherent discourses like (a)-(c) @xcite . the present paper can be viewed as an empirical response to these arguments. it is true that current nlms do not reliably output coherent descriptions when trained on data like (a)-(c). but from text alone, even these imperfect nlms appear to learn implicit models of meaning that are translatable into formal state representations like (a )-(b ). these state representations capture information like the emptiness of the chest in (b ), which is not explicitly mentioned and cannot be derived from any purely syntactic representation of (a)-(b), but follows as a semantically necessary consequence. these implicit semantic models are roughly analogous to the simplest components of discourse representation theory and related formalisms: they represent sets of entities, and update the facts that are known about these entities as sentences are added to a discourse. like the nlms that produce them, these implicit models are approximate and errorprone. nonetheless, they do most of the things we expect of world models in formal semantics: they are structured, queryable and manipulable. in this narrow sense, nlm training appears to induce not just models of linguistic form, but models of meaning. this paper begins with a review of existing approaches to nlm probing and discourse representation that serve as a foundation for our approach. we then formalize a procedure for determining whether nlm representations encode representations of situations like fig. 1 (a )-(b ). finally, we apply this approach to bart and t5 nlms trained on text from the english-language alchemy and textworld datasets. in all cases, we find evidence of implicit meaning representations that: ## approach overview we train probing models to test whether nlms represent the information states specified by the input text. we specifically probe for the truth values of logical propositions about entities mentioned in the text. for example, in figure 1 , we test whether a representation of sentences (a)-(b) encodes the fact that empty(chest) is true and contains(chest, key) is false. ## experiments our experiments aim to discover to what extent (and in what manner) information states are encoded in nlm representations. we first present a specific instantiation of the probe that allows us to determine how well information states are encoded in two nlms and two datasets ( §4.2); then provide a more detailed look at where specific propositions are encoded by varying loc ( §4.3). finally, we describe an experiment investigating the causal role of semantic representations by directly manipulating e(x) ( §4.4). 5 ## conclusion even when trained only on language data, nlms encode simple representations of meaning. in experiments on two domains, internal representations of text produced by two pretrained language models can be mapped, using a linear probe, to representations of the state of the world described by the text. these internal representations are structured, interpretably localized, and editable. this finding has important implications for research aimed at improving factuality and and coherence in nlms: future work might probe lms for the the states and properties ascribed to entities the first time they are mentioned (which may reveal biases learned from training data; @xcite , or correct errors in generation by directly editing representations.