Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distraction (2024)

Abstract: Large language model (LLM) agents show promise in an increasing number of domains. In many proposed applications, it is expected that the agent reasons over accumulated experience presented in an input prompt. We propose the OEDD (Operationalize Experience Despite Distraction) corpus, a human-annotator-validated body of scenarios with pre-scripted agent histories where the agent must make a decision based on disparate experiential information in the presence of a distractor. We evaluate three state-of-the-art LLMs (GPT-3.5 Turbo, GPT-4o, and Gemini 1.5 Pro) using a minimal chain-of-thought prompting strategy and observe that when (1) the input context contains over 1,615 tokens of historical interactions, (2) a crucially decision-informing premise is the rightful conclusion over two disparate environment premises, and (3) a trivial, but distracting red herring fact follows, all LLMs perform worse than random choice at selecting the better of two actions. Our code and test corpus are publicly available at: [github.com/sonnygeorge/OEDD](github.com/sonnygeorge/OEDD).

## introduction a large language model (llm) agent is a system that leverages the world understanding implicit in language generation to produce textual outputs that get interpreted as actions @xcite . many speculate these systems will have a substantial impact and vary from controlling software-e.g., operating a web browser-to manipulating fully embodied robots @xcite . in many and especially long-horizon cases, it is expected that the agent reasons over accumulated experience presented in an input prompt. when such a context of past experience contains grounds for a course of action, the reasoning required to make such an inference often only requires considering one premise. this work asks the question, how do llms behave when the inference required to identify the better of two action paths requires (1) reasoning over two disparate environment facts and (2) not being thrown off by a trivial, but distracting red herring. we implement a novel testing strategy to create the oedd (operationalize experience despite distraction) corpus foot_0 . unlike multi-step task completion tests, oedd tests focus on singular action inference steps. in these tests, the agent is given pre-scripted histories of previous actions and observations as context. by taking this differentiated approach, we are able to control the otherwise environment-accrued pretext and create agent scenarios that require the aforementioned inferencing. using paid human annotators, we refined 16 scenarios in which an agent must make a decision based on disparate information in the presence of a red herring. we expanded these scenarios into temporally independent episodes that stitch together into 12 different reasoning configurations for a total of 192 unique test configurations across all scenarios. ## related work a number of notable existing works aim to probe reasoning in llms by asking questions that require operationalizing context information and are hard to solve through pattern memorization @xcite @xcite . such works do not focus the emulation of real-world-approximating agent scenarios. many works examine multi-hop reasoning @xcite @xcite @xcite , which refers to inferencing that requires more than one inference step and often necessitates the consideration of disparate premises. however, these precedents do not examine this as a precursor to inducing good action choices. lastly, notable work also exists to examine the useful task-completion capabilities of llm agents across many action-inference steps @xcite @xcite @xcite @xcite @xcite @xcite . however, these tests are not built to examine the impacts of variations in action-history contexts on single action-inference steps. to our knowledge at the time of submission, we are the first to do this. balancing novelty and plausibility. while coming up with the information in tests, we aimed to balance both novelty and plausibility as follows: ## experiments for our experiments, we used the prompt templates shown in appendix a.3 to stitch together episodes into environment interaction histories and elicit action-inferences from gpt-3.5-turbo foot_5 , gpt-4o foot_6 , and gemini-1.5-pro foot_7 . notably, these prompt templates do not ask only for the action selection, but ask that the llm indicate its action selection after elaborating a reasoning statement. this is coherent with previous evidence that eliciting a chain of thought before providing a final answer generally enhances performance @xcite . for each of the 12 unique configurations (reasoning/length setups), we extrapolated the probability that a model would select the better choice by sampling between 30 and 50 inferences for each of the 16 tests using the openai foot_8 and google ai foot_9 apis. we stopped sampling for a test early (after the minimum of 30 samples and before the max of 50 samples) when the width of the 90% confidence interval for the true probability of correctly answering was below 0.1. for all models, no configuration had less than 480 (16 × 30) samples taken. in all cases, a model temperature of 0.4 was used. as was our design, to control for the effects of episode order and relative location of crucial details, we reshuffled the order of the historical episodes preceding the final episode before every sample was taken. after extrapolating a model's probability of selecting the better choice for each test in every configuration, we inferred the final configuration probability estimates by averaging these values across the 16 tests. these values are shown in figure 2 . these can be thought of as accuracy scores per configuration. ## discussion our results highlight persisting flaws in current state-of-the-art llms when prompted for agentive action-inference. besides observing degrading performance as prompt size increases, we observe that the llms are consistently thrown off the scent of crucial details by the red herring facts appearing just before the elicited action inference. this is especially true when the crucially informative factors can only be derived by reasoning over two disparate in-context facts. given our tests and minimal zero-shot chain-ofthought prompting strategy, gpt-4o and gemini 1.5 pro only reliably select the better action when the crucially-informative premise is clearly given in a single location and no other, more recent information encroaches upon its topic space with trivial distraction (the 1p setup in furthermore, when (1) the crucially-informative premise can only be derived from two disparate facts, (2) the facts are followed by a trivial distractor, and (3) the prompt length is greater than 1615 tokens, gpt-4o and gemini 1.5 did worse at choosing between two actions than random choice. overall, gemini 1.5 pro performed marginally better than gpt-4o, and gpt-3.5-turbo performed significantly worse than gpt-4o and gemini 1.5. complementing the informativity of other existing and developing benchmarks, we believe that the oedd corpus of agentive reasoning tests stands as a unique, useful, and difficult benchmark for llms and llm agent systems going forward. confounding factors. it is possible that despite our annotation process and careful scrutiny, some of the material introduces unwanted confounding information. thus, we encourage anyone to submit feedback following the process outlined in the readme.md of our public codebase. we hope to maintain a living corpus with continual version updates. prompting. we only tested one prompting strategy. it is possible that other generalized prompting strategies could elicit better results. hence, we provide the corpus publicly for others. bias. we believe that there are likely negative biases in the corpus due to, for example, an underrepresentation of perspectives and lifestyles outside of our spheres of experience. both the writers and annotators of the tests are caucasian males that are professionals in the field of computer science living in the united states. we acknowledge the likely biases as a potential risk and do not recom-mend relying on this corpus as a sole data point for evaluating llm systems. ## future work one of the aims of this work is to establish a preliminary set of results with a simple zero-shot prompting strategy that could, in theory, be improved upon with different prompting techniques or even retrieval augmented generation (rag).