Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models (2023)

Abstract: Recent work studies the cognitive capabilities of language models through psychological tests designed for humans. While these studies are helpful for understanding the general capabilities of these models, there is no guarantee that a model possessing sufficient capabilities to pass those tests would actually use those capabilities in performing real-life tasks. In this work, we formulate task-oriented cognitive capabilities, which are human-like cognitive capabilities that language models leverage to perform tasks. These capabilities are (i) the ability to quickly generate good candidate utterances (the search capability) (ii) the ability to predict how a listener interprets those utterances and choose the most appropriate one (the pragmatic capability). We design an evaluation scheme for comparing these capabilities of a language model with those of a human. Applying this scheme to examine various models in a navigation instruction generation problem, we find that their pragmatic capability is severely lacking. This insight leads us to augment them with better models of the listener and obtain a significant boost of 11% in success rate in guiding real humans. Our work advocates for having a principled procedure for aligning language models with humans that involves (i) formulating task-oriented capabilities, (ii) devising a method to quantify their deficiency, and (iii) iteratively improving them.

## introduction to communicate successfully with humans, language models must possess cognitive capabilities similar to those that facilitate human communication. examining the cognitive capabilities of language models is notoriously challenging because the operations of these models are largely unintelligible to humans. psychologists faced similar challenges when investigating human cognition, and have devised various behavioral tests * the first two authors contribute equally. to diagnose human cognitive capabilities @xcite @xcite . recent work @xcite @xcite applies these tests to evaluate large language models by inputting the tests to these models as prompts and verifying whether they behave like a normal human would. while this approach is helpful for understanding the general limitations of language models, it has two potential drawbacks. first, it is applicable to only large language models that can comprehend human-written prompts, entangling linguistic capability with reasoning capability. second, it shows that a language model can or cannot demonstrate certain mental skills, but does not imply that the model would employ those skills to perform a downstream task. for example, passing false-belief tests does not guarantee that a model will reason about the interpretation of the readers when generating summaries. in general, scoring high on psychological tests may not be sufficient to ensure language models would behave like humans in real-life scenarios. in this work, we take a different approach to evaluating the cognitive capabilities of language models. we define and evaluate task-oriented cognitive capabilities, which are human-like capabilities that a model actually employs to perform the task it is designed for. enhancing these capabilities thus warrants improved performance on the task. to identify these capabilities, we build on two lines of work from socio-cognitive science: bayesian models of cooperative communication @xcite @xcite and studies on how humans implement bayesian reasoning @xcite @xcite . we propose a mathematical cognitive model called bounded pragmatic speaker, which can reasonably characterize the rea-soning processes of both humans and language models. casting humans and language models in the same way enables us to juxtapose their cognitive capabilities. we mathematically formulate two capabilities that a bounded pragmatic agent must possess in order to generate optimally pragmatic utterances. these conditions correspond to wellknown cognitive capabilities of humans: (i) the ability to efficiently generate relevant utterances (the search capability) @xcite @xcite and (ii) the ability to accurately simulate the listener's interpretations of their utterances (the pragmatic capability) @xcite @xcite @xcite . we design a simple procedure to quantitatively evaluate these capabilities of a language model. to evaluate each capability, we compute the task performance gap between the model and an oracle model, which is identical except that the evaluated capability of this model is at human level. figure 1 illustrates our procedure, which theoretically can be applied to any language model. we evaluate various language models on a navigation instruction generation problem @xcite , where a model generates english instructions to guide real humans in photo-realistic 3d environments. foot_0 our evaluation reveals an interesting finding: all evaluated agents possess relatively efficient search capability but inadequate pragmatic capability. we improve the pragmatic capability of the evaluated models by enabling them to reason probabilistically about human listeners @xcite , employing state-of-the-art instructionfollowing agents @xcite @xcite as models of human listeners. we obtain significant improvement in success rate over the original agents, shrinking the gap with human performance on held-out data by 36%. towards eliminating the remaining gap, we illustrate with empirical evidence a major challenge in developing better listener models. specifically, when the instruction-following agents are employed as listener models for the instruction-generating agent, they are required to evaluate ai-generated instructions, which may be significantly different from human-generated • large 𝚫 search , small 𝚫 pragmatic ⇒ improve inference algorithm figure 1 : we propose a framework called bounded pragmatic speaker which can characterize pragmatic reasoning in both humans and language models (a). a bounded pragmatic speaker is composed of a base speaker s base , representing prior knowledge that helps generate instructions efficiently, and a theory-of-mind (tom) listener l tom , a hypothetical model of how the real listener interprets instructions. viewing language models and humans through this unifying lens enables comparing their cognitive capabilities (b). to evaluate a capability of a model, we compare it with an oracle model which is identical except that the evaluated capability is at human level. the outcome of our evaluation can better inform the future direction for improving the model (c). ## related work navigation instruction generation. instruction generation has been commonly studied in navigation settings @xcite @xcite @xcite . the matterport3d simulator and the accompanying datasets (r2r @xcite , r4r @xcite , and rxr @xcite ) offer more challenging settings by combining photo-realistic scenes with long, verbally rich instructions. recent work on evaluating instruction generation agents @xcite reveals the ineffectiveness of standard learning and modeling approaches to this problem. @xcite improve the accuracy and interpretability of instructions in the rxr setting. @xcite leverage this model to synthesize additional data for training instructionfollowing agents. our work aim to offer useful principles to further improve these models. ## problem setting we are concerned with instruction generation: learning a speaker agent r that generates language instructions to guide a human listener h to reach states in an environment. human listener. we imagine a human listener h acting in a partially observed environment with states s. the human does not have access to states but only observations o h and takes actions a h . an instruction @xmath0. a trajectory @xmath1, o 1 , a 1 , • • • , s t , o t , a t ) is an execution of an instruction. the human can follow instructions to generate trajectories in the environment. for example, in an indoor navigation setting, upon hearing "go the kitchen and stop next to the oven", a human walks to the specified location. we define l h (e | u) as the probability that the human generates e upon hearing u. ## task-oriented cognitive capabilities human have evolved highly effective cognitive capabilities for communication. how can we endow a speaker agent with similar capabilities and quantify the degree of resemblance between its capabilities and those of a human? we propose a mathematical framework that reasonably characterizes the human cognitive process for instruction generation ( §4.1). we show that this model can also describe the operation of language models, which allows us to compare them with humans on specific cognitive capabilities. we identify two capabilities that are requisite for any agent implementing our framework to generate optimal instructions ( § 4.2), and introduce an evaluation scheme for collating these capabilities ( §4.3). ## improving pragmatic capability with ensemble instruction-following agents in cases where our evaluation scheme indicates that the pragmatic capability of a language model is deficient, we improve it by installing a better tom listener model. a common approach to learning this listener model is to use the same dataset used for learning the speaker model @xcite . we argue that this approach has a potential drawback. a tom model learned in this way is only exposed to humangenerated input instructions. at deployment time, it would likely experience a covariate shift because as a tom model, the model is then asked to score instructions generated by a speaker model, not by humans. these instructions may be incorrect, ungrammatical, or may simply have a different style than human-generated instructions. this covariate shift would hamper the model's judgement. our preliminary experiments (appendix §a.6) confirm that using a listener trained on only human-generated inputs as the tom model hurts rather than improves the performance of various speakers. we show that this problem can be alleviated by employing tom models that have calibrated uncertainty on unseen instructions. we obtain calibrated models through ensembling @xcite : we train listener models l(k) (e | u), @xmath2. . . k, each on a random 90% subset of the training data with different random initial seeds. we also leverage access to a simulation of the environment to construct better tom models. note that the probability that a tom model l tom assigns to an instruction can be seen as an expectation of a binary metric: ## experimental setup environment and dataset. we employ matter-port3d @xcite , a photo-realistic simulator of the visual perception of a person walking in an indoor environment. at any location, an agent is provided with rgb images capturing the 360-degree panoramic view when looking from that location. we train and evaluate our models using the room-to-room (r2r) language-based navigation dataset. each data point was collected by asking an english-speaking crowd-worker to write a verbal description of a path in an environment. the dataset is split into a training set (61 environments, 4,675 paths), a seen validation set (environments seen during training, 340 paths), and an unseen validation set (11 environments unseen during training, 783 paths). we train the models using the training set and perform model selection on the unseen validation set. performance metrics are computed on the seen validation set. speaker models. we evaluate three speaker architectures: (1) a decoder-only gpt-2 pre-trained on text @xcite ; (2) an lstm encoder-decoder @xcite ; (3) a transformer encoder-decoder @xcite . parameters of the latter two models are randomly initialized. details are in appendix §a.2. ## experiments we investigate the following questions: using our evaluation scheme, we identify that the speakers possess decent search capability but inadequate pragmatic capability. (c) can we improve the speakers by equipping them with better tom listeners? we employ ensembles of state-of-the-art instructionfollowing agents as tom listeners for the speakers, and obtain significant improvements. (d) what are the challenges in bridging the performance gap with human speakers? we show that instruction-following agents trained with only human-generated instructions are not optimal for serving as tom listener models. how well do the speakers perform on our problem? as seen in figure 3 , there is a wide margin between the agent speakers and the human +6.7 +1.2 +3.9 +35.2 +30.9 +25.8 20 30 40 50 60 70 80 fine-tuned gpt-2 encdec-lstm encdec-transformer ndtw original with human-level search with human-level tom humans (r2r annotators) figure 4: performance (in ndtw) of the speakers and their human-augmented versions. possessing humanlevel pragmatic capability improves performance of the speakers, showing that their original pragmatic capability is highly deficient compared to that of a human. what causes the speakers' deficiency? next, we investigate whether the lack of search or pragmatic capability is responsible for the deficiency of the speakers. the prospective performance gains presented in figure 4 show that it is under-performed pragmatic capability that primarily causes the models to perform poorly. specifically, while equipping the models with oracle search capability only improves their performance by 9.4% on average, granting them oracle pragmatic capability nearly doubles their performance metrics. in fact, the search capability of the models is already as good as that of the humans we employ, because the models with oracle pragmatic capabil-ity achieve even slightly higher ndtw scores than the human speakers. can we improve the speakers by equipping them with better tom models? following the procedure described in section §5, we train state-of-theart instruction-following agents to serve as tom listener models for the speakers. performances of different combinations of speakers and listeners are given in what are the challenges in bridging the performance gap with human speakers? despite the promising improvements, there remains a large gap of 17.9 ndtw points between our best speaker and the human speakers. as suggested by figure 4 , this gap can be closed by developing accurate tom models. we argue that optimal tom models cannot be simply obtained by learning optimal instructionfollowing agents, because the latter is learned to execute human-generated instructions while the former is asked to rank model-generated instructions. to illustrate the difference, we measure the agreement between human and model listeners on instructions generated by different speakers. we define the agreement score between a human l h and a model l as as seen in ## conclusion this work introduces a framework for analyzing task-oriented cognitive capabilities of instructiongeneration language models. we show that insights from the analysis are helpful in directing development on these models. our results highlight the necessity of constructing better tom models for improving these models. we argue that learning accurate tom listener models is met with novel, distinct challenges. we hope that our findings will motivate the community to focus more on evaluating task-oriented cognitive capabilities and to create datasets, training methods, and evaluation procedures for enhancing the pragmatic capability of language models.