Prospector: Improving LLM Agents with Self-Asking and Trajectory Ranking (2024)

Abstract: Large language models (LLMs) have shown the ability to solve complex decision-making tasks beyond natural language processing tasks. LLM agents based on few-shot in-context learning (ICL) achieve surprisingly high performance without training. Despite their simplicity and generalizability, ICL-based agents are limited in their ability to incorporate feedback from an environment. In this paper, we introduce Prospector, an LLM agent that consists of two complementary LLMs, an Actor and a Critic. To elicit better instruction-aligned actions from the LLM agent, we propose AskAct prompting that performs an additional self-asking step such as goal and progress checking before generating an action. Furthermore, to implicitly incorporate the environment feedback, we propose Trajectory Ranking that orders generated trajectories by predicting the expected total reward. Prospector encourages the LLM Actor to generate diverse (creative) trajectories, and harnesses the LLM Critic to select the most rewarding trajectory. On representative decision-making benchmark environments such as ALFWorld and WebShop, we empirically demonstrate that Prospector can considerably increase the success rate of given tasks, while outperforming recent advancements such as ReAct and Reflexion.

## introduction large language models (llms) @xcite @xcite @xcite have achieved remarkable success in natural language processing (nlp). recently, llm-based autonomous agents @xcite @xcite @xcite have shown the ability to solve complex decision-making tasks such as householding @xcite , science experiments @xcite , web navigation @xcite , computer control @xcite , figure 1 : high level comparison of llm agent approaches. we compare three main approaches: (1) in-context learning (icl)-based llm agents, (2) finetuning (ft)-based llm agents, and (3) complementary llm agents (ours). prospector is designed to combine the advantages of the both worlds: in-context learning and fine-tuning. prospector consists of two complementary llms such as a very large llm actor and a relatively small llm critic. icl-based llm agents such as react @xcite can solve interactive decision-making tasks by imitating the few-shot demonstrations given in the prompt. they achieve surprisingly high performance without training. despite the simplicity and generalizability, the icl-based approaches lack optimizing trajectories based on the reward from an environment. in contrast, ft-based llm agents such as fireact @xcite and @xcite have been recently introduced. even though they can achieve high performance on a target task by fine-tuning the llm agent based on environment interaction, they may not be gener-ally applicable to diverse tasks due to the out-ofdistribution (ood) problem. in this paper, we introduce prospector, a llm agent that consists of two complementary llms such as llm actor and llm critic. as shown in figure 1 , prospector is designed to combine the advantages of the both worlds: in-context learning and fine-tuning. prospector uses a very large llm (gpt-3-175b @xcite or llama-2-70b @xcite ) for the llm actor, and the llm actor generates a language action for a given language observation by leveraging the ability of few-shot in-context learning of llms. to align the llm actor on an environment, prospector uses a relatively small llm as the llm critic that can rank candidate trajectories by predicting the expected total reward from the given environment. the llm critic is fine-tuned on trajectoryreward paired data to precisely reflect the rewards from the environment. we would like to note that prospector can not only maintain the strong generality of the llm actor by using the ability of in-context learning, but also effectively reflect rewards from an environment by fine-tuning the llm critic. also, we would like to highlight that simply adding a llm critic fine-tuned on a specific environment, prospector can effectively expand decision-making tasks that can be solved without fine-tuning the llm actor with huge parameters. more specifically, to improve icl-based llm agents, prospector provides two techniques: (1) askact prompting for llm actor, and (2) trajectory ranking (tr) for llm critic. first, even though icl-based llm agents such as react provide a surprisingly high success rate, it can suffer from hallucination such as unaligned action generation. to alleviate hallucination and elicit better instruction-aligned actions from the llm actor, we propose askact prompting that performs an additional self-asking step such as goal and progress checking before generating an action. this allows the llm actor to collect the information necessary for decision-making by itself, and help to elicit better instruction-aligned actions than reason-only prompting such as react. second, icl-based llm agents lack reflecting environment feedback when generating actions. to implicitly incorporate environment feedback, we provide trajectory ranking (tr) that can order generated trajectories by predicting the expected total reward. prospector encourages the llm actor to generate diverse (creative) trajectories at high temperature, and harnesses the llm critic to select the most rewarding trajectory among the candidates. since fine-tuning a llm agent from environment feedback is often prohibitive due to its enormous training cost, we consider fine-tuning a relatively small llm critic by constructing a (trajectory, reward) dataset. prospector can achieve high performance through the synergy of askact prompting, which can generate promising trajectory candidates, and trajectory ranking, which can select the most rewarding trajectory from the candidates. in the experiments, we demonstrate that prospector outperforms recent advancements such as re-act @xcite and reflexion @xcite on the representative language-based interactive decision-making benchmarks including alf-world @xcite and webshop @xcite . the contributions of this paper can be summarized as follows: ## method the overview of prospector is shown in figure 2 . prospector consists of two complementary llms such as the llm actor and llm critic. the llm actor generates a text-based action a t conditioned on a few-shot demonstration τ d 1: prospector is a llm agent that consists of two complementary llms such as llm actor and llm critic for solving complex interactive decision-making tasks. llm actor generates actions conditioned on the few-shot demonstrations and the history of observations and actions. to elicit the more proper actions, prospector llm actor uses askact prompting that interleaves self-asking steps in the few-shot demonstrations. furthermore, to implicitly incorporate environment feedback, we introduce trajectory ranking which harnesses llm critic that can select the most rewarding trajectories by predicting the expected total reward. {o 1 , k 1 , a 1 , ..., o t-1 , k t-1 , a t-1 }, a current observation o t , and a current think k t . formally, it can be denoted as a t ∼ π θ (a t |τ d 1:t , τ 1:t-1 , o t , k t ), where π θ denotes the probability distribution induced by the llm actor. unlike react-based llm agents, prospector llm actor uses askact prompting. we introduce askact prompting to alleviate the hallucination of react prompting and elicit better actions from the llm. we provide the details of askact prompting in the following subsection. one of the key ideas of prospector is to encourage llm actor to generate diverse (creative) trajectories and leverage llm critic to select the most rewarding trajectory among generated trajectories. the llm critic is responsible for performing trajectory ranking (tr) that orders trajectories {τ i } n @xmath0. more concretely, the ranked ordering of the trajectories is given by (τ 1 , ..., τ n ) such that ( @xmath1... > rn ). we provide more details of trajectory ranking in the following subsection. ## conclusion we introduce prospector, a llm agent that consists two complementary llm agents, llm actor and llm critic. to improve the baseline performance of llm actor, we provide askact prompting that introduces additional self-asking steps in the few-shot demonstrations. also, to implicitly reflect environment feedback, we provide trajectory ranking that selects the most rewarding trajectory by predicting the expected total reward. evaluation on representative decision-making benchmark environments such as alfworld and webshop shows that prospector can considerably increase the success rate of given tasks, while outperforming recent icl-based agents such as react and reflexion.