Language Modeling with Latent Situations (2023)

Abstract: Language models (LMs) often generate incoherent outputs: they refer to events and entity states that are incompatible with the state of the world described in inputs. We introduce SITUATIONSUPERVISION, a family of approaches for improving coherence in LMs by training them to construct and condition on explicit representations of entities and their states. SITUATIONSUPERVISION has two components: an *auxiliary situation modeling* task that trains models to predict entity state representations in context, and a *latent state inference* procedure that imputes these states from partially annotated training data. SITUATIONSUPERVISION can be applied via fine-tuning (by supervising LMs to encode state variables in their hidden representations) and prompting (by inducing LMs to interleave textual descriptions of entity states with output text). In both cases, it requires only a small number of state annotations to produce substantial coherence improvements (up to an 16% reduction in errors), showing that standard LMs can be efficiently adapted to explicitly model language and aspects of its meaning.

## introduction recent years have seen dramatic improvements in the quality of text generated by neural language models (lms; @xcite . nevertheless, even the best lms still suffer from failures of semantic coherence. samples from lms refer to entities that have not yet been mentioned, assert contradictory facts, or describe impossible sequences of events @xcite . this paper introduces situationsupervi-sion, a family of methods for efficiently mitigating incoherent language generation. situation-supervision adapts pre-trained lms to explicitly sam unzipped the suitcase. ## background and preliminaries a language model (lm) encodes a distribution p(t ′ | t ) over texts t ′ given contexts t (fig. 1 ). today, most lms are implemented as deep neural networks trained on massive text corpora @xcite . sampling from them produces naturalistic text that often resembles human-generated language. however, lm generation is prone to several failure modes, including generation of text that is incoherent, untruthful, or unreliable @xcite @xcite . past work has shown that some of these behaviors stem from models' failure to build good representations, both of entities' default properties @xcite and state changes in context @xcite . humans' ability to avoid these failure modes, and to generate truthful and coherent text, is generally understood to rest upon explicit mental representations of the situations that language communicates. the nature and structure of these representations remains an ongoing topic of research in linguistics and cognitive science, but existing theories broadly agree that language users maintain explicit beliefs about the properties of and relations among entities mentioned in a discourse, updating these beliefs in response to new observations or new information conveyed in language (e.g. @xcite . these representational theories suggest that language models p(t ′ | t ) may also benefit from explicit modeling of situation state. given an input text t , we wish to first represent the situation s described by t before predicting a next sentence. inspired by models of situation semantics in the linguistics literature @xcite we propose to model situations as sets of propositions s i that are known or inferable about entities in a discourse. 2 examples, with propositions expressed as sentences in natural language, are shown in fig. 1(b) and fig. 2 . having inferred s from t , we may then condition on it when sampling t ′ from p(t ′ | s, t ). past work has proposed a number of language generation models that explicitly model the state of the world, primarily by developing specialized prediction architectures that maintain internal state representations @xcite or interact with outside simulation engines @xcite . while effective, these approaches come at a cost-requiring complex training data @xcite , limiting models to narrow, pre-defined domains, and generally precluding the large-scale (text-only) pretraining responsible for many of the greatest successes of current lms. the main question this paper seeks to answer is whether the benefits of explicit world modeling may be obtained entirely within the language modeling paradigm itself, without specialized model architectures or large amounts of specialized supervision. we do so by adapting pre-trained lms to better represent situations s. there are two standard frameworks for lm adaptation. in smaller models, which are generally adapted by fine-tuning of model parameters, we develop auxiliary loss functions that encourage models' hidden states to contain the information required to generate textual descriptions of state. 3 in larger models, which can also be prompted by prepending a task description or set of examples to the input context, we develop prompts that induce models to generate textual state descriptions in lm output itself. our research builds on a large body of work that uses auxiliary prediction tasks to shape model representations, notably work using "scaffold" decoders to shape model representations of syntax @xcite , and and "scratchpad" or "chain-of-thought" approaches to perform intermediate computations in models' output spaces @xcite @xcite . in §3, we show how to adapt both techniques for a new class of text generation problems. adapting lms with auxiliary prediction tasks requires a source of data for auxiliary supervision. this kind of supervision is uniquely difficult to obtain for generation tasks. but the probabilistic framing described above makes it natural to formulate language modeling with explicit situations as a latent variable problem. at training time, we may use context t and targets t ′ to guide inference of the unknown s from which t ′ was predicted. once inferred, these states supervise the representation-building model that predicts s from t alone. as above, a great deal of past work has focused on treating string-valued prompts or plans as latent variables @xcite @xcite . in §4, we generalize these methods to support multi-step text generation, and show that inferred states can be used to supervise small models as well as prompt large ones. ## auxiliary situation modeling we begin by assuming access to a pre-trained lm and two sources of supervision: a dataset x u of text examples of the form (t, t ′ ), and a smaller dataset x a of examples (t, s, t ′ ) annotated with textual situation descriptions s. our full training data x is thus x u ∪ x a . as depicted in fig. 2 , we take these situation descriptions to consist of declarative sentences about the properties and relations of entities that are relevant to the text being generated. in this section, we describe two auxiliary prediction schemes that use these annotations to improve the lm's ability to model the conditional text distribution p(t ′ | t ). ## latent state inference the methods described in §3 applied situa-tionsupervision only to examples for which a ground-truth state annotation was provided. for these methods to be effective, enough state annotations must be available to provide a useful training signal in the auxiliary loss or to specify the auxiliary prediction task for the prompted model. but such state annotations are in general both hard to collect and hard to design. in this section we describe how to obtain them automatically, without the need for large amounts of annotation. below, we re-formulate the two approaches in §3 as latent variable models that can infer and condition on state representations even for unannotated training documents. intuitively, this inference problem is easier at training time than prediction time: knowing what text followed a context constrains the possible situations the context could describe. most work on semisupervised inference of auxiliary prediction targets has focused on automatic optimization of prompts and reasoning chains @xcite . to the best of our knowledge, inferred latent variables have not been used to train auxiliary decoders or to design intermediate state representation for multi-step text generation. the techniques described below are quite general, and might be productively employed beyond the generation applications we describe here. ## experimental setup datasets we evaluate situationsupervision on english language modeling datasets. tw is a set of 1368 transcripts (992 train / 376 evaluation) derived from textworld @xcite we generate a set of textual game transcripts where players navigate through a house, unlocking doors and containers to hunt for a target object. the lm is trained on these transcripts to generate next actions. as state supervision, we use the set of state variables (given as entity-centric facts) that are known and relevant in the current context (see §7.1 for more details). trip @xcite ) is a set of 1643 plausible and implausible fivesentence stories (1169 train / 474 evaluation) which require physical commonsense reasoning to disambiguate. models are trained to generate judgments of whether or not a given sentence is acceptable in a context. the state is given by a set of attributes for each entity, which is updated after each sentence. 4 each passage @xmath0, t ′ 1 , • • • , t ′ n . in tw, each chunk consists of a textual action description followed by a game response. in trip, each chunk is a single sentence from the story followed by a plausibility judgment. we test coherence of generating each t ′ from its context t . for the annotated passages in 4 see appendix a for state representation details. x a , each context t i is annotated with corresponding state information s i . thus, passages in x u can be broken down into (t, t ′ ) pairs, while passages in x a can be broken down into (t, s, t ′ ) triples. models for fine-tuning experiments, we use bart-base @xcite ) as the language model and fine-tune it using the adamw optimizer with learning rate 1e-5, stopping once validation accuracy has stopped improving for 10 epochs. for prompting experiments, we use the gpt3 da-vinci-002 model @xcite ). 5 metrics to evaluate models on tw, we sample next actions from the lm and compute the fraction of these that are semantically coherent using the tw simulator. 67 for trip, we evaluate every story pair by training models to predict the string ok or not ok after each sentence depending on whether it is semantically acceptable within a given context. the trip dataset contains human semantic acceptability judgments for each sentence of each stories; we evaluate the accuracy with which models predict these acceptability judgments (labeling a story as unacceptable if any sentence is predicted to be unacceptable). for tw, we report sentence-wise metrics: we measure the fraction of next sentences which are generated to be coherent within the context. in trip, we report passage-wise metrics: we measure the percent of complete passages for which every sentence of the passage is labelled accurately. as baselines in each domain, we compare to ordinary fine-tuning and prompting. as far as we are aware, no prior work in these domains focus on evaluating generation coherence or accuracy. vision with 500 seed states improves accuracy by ∼ 7% over a text-only baseline. note in this case that the latent inference procedure was unable to improve beyond auxiliary training. however, even adding in the remaining 500 ground-truth state annotations does not improve the lm, indicating that perhaps the 500 seed states were sufficient for the lm to learn everything it can from state supervision. ## conclusion effective generation of coherent text requires reasoning about the world that text describes. in this work, we use entity states as auxiliary supervision to improve lms ability to perform this reasoning under both fine-tuning and prompting. we find that when either annotation budget (for fine-tuning) or context window size (for prompting) are limited, it is more sample-and token-efficient to increase the amount of state supervision rather than text-only supervision. however, since state annotations are harder to collect, we introduce latent supervision algorithms for sample-efficiently improving lm generation coherence, and demonstrate improvements in two domains. our results point to a potentially broad role for semantic supervision in lm training and prompting-even small amounts can yield large coherence improvements. this work more generally suggests that semantic state reasoning is still challenging for even modern large language models, and but can be improved without fundamental changes to the architecture of existing lms.