Q u BE : Question-based Belief Enhancement for Agentic LLM Reasoning (2024)

Abstract: Despite advancements in Large Language Models (LLMs), many complex tasks are not easily solved in a single inference step, requiring the use of agentic LLMs in interactive environments. However, agentic LLMs suffer from a phenomenon known as reasoning derailment, due to the indiscriminate incorporation of observations from partially observable environments. We introduce QuBE, a method that enhances agents’ focus on task-relevant contexts, by constructing a belief state via question answering. We validate QuBE through experiments in two agentic LLM scenarios with partial observability: 1) a canonical interactive decision-making scenario using text-based game engines, and 2) an interactive retrieval-augmented generation (RAG) scenario using search engines. In the AlfWorld text-based game, QuBE outperforms established baselines by substantial margins, and in the search engine scenario, it achieves marked improvements on the BeIR zero-shot retrieval benchmark. The results demonstrate that QuBE significantly mitigates reasoning derailment, refining the decision-making process of LLM agents in partially observed environments.

## introduction recent advancements in the general capabilities of large language models (llms) has led to an explosion of interest in their deployment as agents, capable of autonomously performing complex, long-horizon tasks through interactive environments @xcite @xcite . llm-based agents leverage the llm's capability to interleave environment interaction (actions) with reflective reasoning (rationales) @xcite @xcite , enabling sophisticated action policies for complex tasks. despite these advances, llm-based agents still face a critical issue known as reasoning derailment @xcite , a phenomenon in which agents struggle to recover from an initial erroneous rationale, resulting in a further cycle of incorrect reasoning and unsuccessful interactions. in this work, we identify the partially observable nature of many interactive environments @xcite @xcite , as a key factor contributing to reasoning derailment. in partially observable environments, observations available to the agent provide only an incomplete view of the underlying, true state of the environment which influences the agent's task. as such, the agent must effectively resolve the uncertainty and incompleteness in the observed information, to optimize its reasoning and actions. however, existing llm agent methods such as react @xcite and reflexion @xcite are ill-equipped to address this challenge, as they operate directly on raw observed states, stored in a non-selective input context. rather, such naive utilization of observations exacerbates the accumulation of noisy, incomplete, and task-irrelevant observed states, making reasoning derailment more likely. we illustrate this phenomenon in fig. 1-(a ). in this alfworld example, the llm agent erroneously generates the action "go to desk 1" despite already being there, which acts as a trigger for subsequent reasoning derailment due to a failure to distinguish between its current location (desk 2) and the target object's location (desk 1). to mitigate reasoning derailment in llm agents, we propose question-based belief enhancement (qube), to dynamically construct a focused, taskrelevant belief state, which estimates the unobserved true state of the environment. specifically, we formulate belief state construction as a questionanswering process, implemented using an llm with access to environment tools. by enabling the llm to direct queries to both the agent's context history, as well as the environment itself, we ensure the construction of a belief state that resolves the ambiguity and noisiness of partial environment observations in a grounded manner. in fig. 1-(b , c ), we illustrate how qube addresses reasoning derailment through the construction of a belief state. using a series of queries, qube constructs a concise, task-relevant belief state (shown in the blue box) which fills in the gaps due to partial observability. in the alfworld task, the belief state includes details like checked receptacles, inferred from the agent's context history (q-context), as well as the agent's current inventory and location, which are issued as queries to the game engine (q-env). next, we repurpose the llm to produce a textual rationale based on the belief state, as shown in the green box. by conditioning on the constructed belief state, the rationale accurately locates the target object-desklamp 1 at desk 1-enabling qube to resolve derailment, and guide the agent's next actions toward task success. we conduct experiments on two unique partial observability scenarios, on a canonical scenario using the text-based game engine, alfworld @xcite , and a challenging, interactive retrieval scenario using a search engine, on the beir benchmark @xcite . in alfworld, qube demonstrates significant improvements over established baselines, react and reflexion, by 28% and 24% absolute in task success rate, respectively. moreover, in the beir benchmark, qube consistently enhances performance, achieving a notable increase of 1.3% in ndcg@10 over react, on the challenging task of zero-shot retrieval. the main contributions of our work are summarized as follows: ## environments in this section, we describe in detail our selection of environments, which aims to represent a diverse selection encompassing different task types and reward availabillity. ## method motivated by our preliminary study, we present our proposed method, qube, in the following section. next, we begin by providing a broad description of standard llm-based agents, within which we discuss the systematic cause of reasoning derailment. ## experiments to validate the effectiveness of qube, we conduct experiments on our two target settings in ge and 4 implementation details of the keyword extraction process are provided in a.6. se. in both scenarios, qube components are implemented using llms, in conjunction with a react llm agent foot_2 ## conclusion in this work, we studied deployment of agentic llms within complex interactive environments. due to the challenge of partial observability, existing llms produce reasoning errors, which are even more pronounced in agentic llms requiring multiple interactions. to tackle this, we propose qube, which constructs a belief state using question-answering, as an estimate of the unobserved true environment state. using the constructed belief state, qube performs rationale generation to resolve noise and uncertainty in the raw partial observations and enhance the llm agent's action policy. experimented on search and game environment scenarios, qube reduces reasoning errors, and improves task performance, outperforming strong baselines.