R E A C T : S YNERGIZING R EASONING AND A CTING IN L ANGUAGE M ODELS
A BSTRACT
While large language models (LLMs) have demonstrated impressive performance across tasks in language understanding and interactive decision making, their abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action plan generation) have primarily been studied as separate topics.
In this paper, we explore the use of LLMs to generate both reasoning traces and task-speciﬁc actions in an interleaved manner, allowing for greater synergy between the two: reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with and gather additional information from external sources such as knowledge bases or environments.
We apply our approach, named ReAct , to a diverse set of language and decision making tasks and demonstrate its effectiveness over state-of-the-art baselines in addition to improved human interpretability and trustworthiness.
Concretely, on question answering (HotpotQA) and fact veriﬁcation (Fever), ReAct overcomes prevalent issues of hallucination and error propagation in chain-of-thought reasoning by interacting with a simple Wikipedia API, and generating human-like task-solving trajectories that are more interpretable than baselines without reasoning traces.
Furthermore, on two interactive decision making benchmarks (ALFWorld and WebShop), ReAct outperforms imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively, while being prompted with only one or two in-context examples.
I NTRODUCTION
A unique feature of human intelligence is the ability to seamlessly combine task-oriented actions with verbal reasoning (or inner speech, Alderson-Day & Fernyhough, 2015), which has been theorized to play an important role in human cognition for enabling self-regulation or strategization (Vygotsky, 1987; Luria, 1965; Fernyhough, 2010) and maintaining a working memory (Baddeley, 1992).
Consider the example of cooking up a dish in the kitchen.
Between any two speciﬁc actions, we may reason in language in order to track progress (“now that everything is cut, I should heat up the pot of water”), to handle exceptions or adjust the plan according to the situation (“I don’t have salt, so let me use soy sauce and pepper instead”), and to realize when external information is needed (“how do I prepare dough?
Let me search on the Internet”).
We may also act (open a cookbook to read the recipe, open the fridge, check ingredients) to support the reasoning and to answer questions (“What dish can I make right now?”).
This tight synergy between “acting” and “reasoning” allows humans to learn new tasks quickly and perform robust decision making or reasoning, even under previously unseen circumstances or facing information uncertainties.
Recent results have hinted at the possibility of combining verbal reasoning with interactive decision making in autonomous systems.
On one hand, properly prompted large language models (LLMs) have demonstrated emergent capabilities to carry out several steps of reasoning traces to derive [Table]
(cid:3)(cid:51)(cid:88)(cid:87)(cid:3)(cid:86)(cid:82)(cid:80)(cid:72)(cid:3)(cid:83)(cid:72)(cid:83)(cid:83)(cid:72)(cid:85)(cid:3)(cid:86)(cid:75)(cid:68)(cid:78)(cid:72)(cid:85)(cid:3)(cid:82)(cid:81)(cid:3)(cid:68)(cid:3)(cid:71)(cid:85)(cid:68)(cid:90)(cid:72)(cid:85)(cid:17) (cid:11)(cid:21)(cid:12)(cid:3)(cid:36)(cid:79)(cid:73)(cid:58)(cid:82)(cid:85)(cid:79)(cid:71) (cid:11)(cid:20)(cid:12)(cid:3)(cid:43)(cid:82)(cid:87)(cid:86)(cid:83)(cid:82)(cid:87)(cid:3)(cid:52)(cid:36) answers from questions in arithmetic, commonsense, and symbolic reasoning tasks (Wei et al., 2022).
However, this “chain-of-thought” reasoning is a static black box, in that the model uses its own internal representations to generate thoughts and is not grounded in the external world, which limits its ability to reason reactively or update its knowledge.
This can lead to issues like fact hallucination and error propagation over the reasoning process (Figure 1 (1b)).
On the other hand, recent work has explored the use of pre-trained language models for planning and acting in interactive environments (Ahn et al., 2022; Nakano et al., 2021; Yao et al., 2020; Huang et al., 2022a), with a focus on predicting actions via language priors.
These approaches usually convert multi-modal observations into text, use a language model to generate domain-speciﬁc actions or plans, and then use a controller to choose or execute them.
However, they do not employ language models to reason abstractly about high-level goals or maintain a working memory to support acting, barring Huang et al.
(2022b) who perform a limited form of verbal reasoning to reiterate spatial facts about the current state.
Beyond such simple embodied tasks to interact with a few blocks, there have not been studies on how reasoning and acting can be combined in a synergistic manner for general task solving, and if such a combination can bring systematic beneﬁts compared to reasoning or acting alone.
In this work, we present ReAct , a general paradigm to combine reasoning and acting with language models for solving diverse language reasoning and decision making tasks (Figure 1).
ReAct prompts LLMs to generate both verbal reasoning traces and actions pertaining to a task in an interleaved manner, which allows the model to perform dynamic reasoning to create, maintain, and adjust high-level plans for acting (reason to act), while also interact with the external environments (e.g. Wikipedia) to incorporate additional information into reasoning (act to reason).
We conduct empirical evaluations of ReAct and state-of-the-art baselines on four diverse benchmarks: question answering (HotPotQA, Yang et al., 2018), fact veriﬁcation (Fever, Thorne et al., 2018), text-based game (ALFWorld, Shridhar et al., 2020b), and webpage navigation (WebShop, Yao et al., 2022).
For HotPotQA and Fever, with access to a Wikipedia API that the model can interact with, ReAct outperforms vanilla action generation models while being competitive with chain-ofthought reasoning ( CoT ) (Wei et al., 2022). The best approach overall is a combination of ReAct and CoT that allows for the use of both internal knowledge and externally obtained information during reasoning.
On ALFWorld and WebShop, two or even one-shot ReAct prompting is able to outperform imitation or reinforcement learning methods trained with 10 3 ∼ 10 5 task instances, with an absolute improvement of 34% and 10% in success rates respectively.
We also demonstrate the importance of sparse, versatile reasoning in decision making by showing consistent advantages over controlled baselines with actions only.
Besides general applicability and performance boost, the combination of reasoning and acting also contributes to model interpretability, trustworthiness, and diagnosability across all domains, as humans can readily distinguish information from model’s internal knowledge versus external environments, as well as inspect reasoning traces to understand the decision basis of model actions.
To summarize, our key contributions are the following: (1) we introduce ReAct , a novel promptbased paradigm to synergize reasoning and acting in language models for general task solving; (2) we perform extensive experiments across diverse benchmarks to showcase the advantage of ReAct in a few-shot learning setup over prior approaches that perform either reasoning or action generation in isolation; (3) we present systematic ablations and analysis to understand the importance of acting in reasoning tasks, and reasoning in interactive tasks; (4) we analyze the limitations of ReAct under the prompting setup (i.e. limited support of reasoning and acting behaviors), and perform initial ﬁnetuning experiments showing the potential of ReAct to improve with additional training data.
Scaling up ReAct to train and operate on more tasks and combining it with complementary paradigms like reinforcement learning could further unlock the potential of large language models to reason and act in interactive settings.
R E A CT : S YNERGIZING R E ASONING + A CT ING
Consider a general setup of an agent interacting with an environment for task solving.
At time step t , an agent receives an observation o t ∈ O from the environment and takes an action a t ∈
A following some policy π ( a t | c t ) , where c t = ( o 1 , a 1 , · · · , o t − 1 , a t − 1 , o t ) is the context to the agent.
Learning a policy is challenging when the mapping c t (cid:55)→ a t is highly implicit and requires extensive computation.
For example, the agent shown in Figure 1(1c) is unable to generate the correct ﬁnal action (Act 4) to ﬁnish the QA task as it requires complex reasoning over the trajectory context (Question, Act 1-3, Obs 1-3).
Similarly, the agent shown in Figure 1(2a) fails to comprehend from the context that sinkbasin 1 does not contain peppershaker 1, thus keep producing hallucinating actions.
The idea of ReAct is simple: we augment the agent’s action space to ˆ A
=
A ∪ L , where L is the space of language.
An action ˆ a t ∈ L in the language space, which we will refer to as a thought or a reasoning trace , does not affect the external environment, thus leading to no observation feedback.
Instead, a thought ˆ a t aims to compose useful information by reasoning over the current context c t , and update the context c t +1 =
( c t , ˆ a t ) to support future reasoning or acting.
As shown in Figure 1, there could be various types of useful thoughts, e.g. decomposing task goals and create action plans (2b, Act 1; 1d, Thought 1), injecting commonsense knowledge relevant to task solving (2b, Act 1), extracting important parts from observations (1d, Thought2, 4), track progress and transit action plans (2b, Act 8), handle exceptions and adjust action plans (1d, Thought 3), and so on.
However, as the language space L is unlimited, learning in this augmented action space is difﬁcult and requires strong language priors.
In this paper, we mainly focus on the setup where a frozen large language model, PaLM-540B (Chowdhery et al., 2022), is prompted with few-shot in-context examples to generate both domain-speciﬁc actions and free-form language thoughts for task solving (Figure 1 (1d), (2b)).
Each in-context example is a human trajectory of actions, thoughts, and environment observations to solve a task instance (see Appendix A).
For the tasks where reasoning is of primary importance (Figure 1(1)), we alternate the generation of thoughts and actions so that the task-solving trajectory consists of multiple thought-action-observation steps.
In contrast, for decision making tasks that potentially involve a large number of actions (Figure 1(2)), thoughts only need to appear sparsely in the most relevant positions of a trajectory, so we let the language model decide the asynchronous occurrence of thoughts and actions for itself.
Since decision making and reasoning capabilities are integrated into a large language model, ReAct enjoys several unique features: Intuitive and easy to design . Designing ReAct prompts is straightforward as human annotators just type down their thoughts in language on top of their actions taken. No ad-hoc format choice, thought design, or example selection is used in this paper.
We detail prompt design for each task in Sections 3 and 4. 2.
General and ﬂexible . Due to the ﬂexible thought space and thought-action occurrence format, ReAct works for diverse tasks with distinct action spaces and reasoning needs, including but not limited to QA, fact veriﬁcation, text game, and web navigation.
Performant and robust .
ReAct shows strong generalization to new task instances while learning solely from one to six in-context examples, consistently outperforming baselines with only reasoning or acting across different domains.
We also show in Section 3 additional beneﬁts when ﬁnetuning is enabled, and in Section 4 how ReAct performance is robust to prompt selections.
Human aligned and controllable . ReAct promises an interpretable sequential decision making and reasoning process where humans can easily inspect reasoning and factual correctness. Moreover, humans can also control or correct the agent behavior on the go by thought editing, as shown in Figure 4 in Section 4.
K NOWLEDGE -I NTENSIVE R EASONING T ASKS
We begin with knowledge-intensive reasoning tasks like multi-hop question answering and fact veriﬁcation.
As shown in Figure 1(1d), by interacting with a Wikipedia API, ReAct is able to retrieve information to support reasoning, while also use reasoning to target what to retrieve next, demonstrating a synergy of reasoning and acting.
3. S ETUP
Domains We consider two datasets challenging knowledge retrieval and reasoning: (1) HotPotQA (Yang et al., 2018), a multi-hop question answering benchmark that requires reasoning over two or more Wikipedia passages, and (2) FEVER
(Thorne et al., 2018), a fact veriﬁcation benchmark where each claim is annotated SUPPORTS, REFUTES, or NOT ENOUGH INFO, based on if there exists a Wikipedia passage to verify the claim.
In this work, we operate in a question-only setup for both tasks, where models only receive the question/claim as input without access to support paragraphs, and have to rely on their internal knowledge or retrieve knowledge via interacting with an external environment to support reasoning.
Action Space We design a simple Wikipedia web API with three types of actions to support interactive information retrieval: (1) search [ entity ], which returns the ﬁrst 5 sentences from the corresponding entity wiki page if it exists, or else suggests top-5 similar entities from the
Wikipedia search engine, (2) lookup [ string ], which would return the next sentence in the page containing string , simulating Ctrl+F functionality on the browser.
(3) finish [ answer ], which would ﬁnish the current task with answer .
We note that this action space mostly can only retrieve a small part of a passage based on exact passage name, which is signiﬁcantly weaker than state-of-theart lexical or neural retrievers.
The purpose is to simulate how humans would interact with Wikipedia, and force models to retrieve via explicit reasoning in language.
3. M ETHODS
ReAct Prompting For HotpotQA and Fever, we randomly select 6 and 3 cases 1 from the training set and manually compose ReAct -format trajectories to use as few-shot exemplars in the prompts.
Similar to Figure 1(d), each trajectory consists of multiple thought-action-observation steps (i.e. dense thought), where free-form thoughts are used for various purposes. Speciﬁcally, we use a combination of thoughts that decompose questions (“I need to search x, ﬁnd y, then ﬁnd z”), extract information from Wikipedia observations (“x was started in 1844”, “The paragraph does not tell x”), perform commonsense (“x is not y, so z must instead be...”) or arithmetic reasoning (“1844 < 1989”), guide search reformulation (“maybe I can search/look up x instead”), and synthesize the ﬁnal answer (“...so the answer is x”).
See Appendix A for more details.
Baselines We systematically ablate ReAct trajectories to build prompts for multiple baselines (with formats as Figure 1(1a-1c)):
(a) Standard prompting ( Standard ), which removes all thoughts, actions, observations in ReAct trajectories.
(b) Chain-of-thought prompting ( CoT ) (Wei et al., 2022), which removes actions and observations and serve as a reasoning-only baseline.
We also build a self-consistency baseline ( CoT-SC )
(Wang et al., 2022a;b) by sampling 21 CoT trajectories with decoding temperature 0.7 during inference and adopting the majority answer, which is found to consistently boost performance over CoT .
(c) Acting-only prompt ( Act ), which removes thoughts in ReAct trajectories, loosely resembling how WebGPT (Nakano et al., 2021) interacts with the
Internet to answer questions, though it operates on a different task and action space, and uses imitation and reinforcement learning instead of prompting.
Combining Internal and External Knowledge As will be detail in Section 3.3, we observe that the problem solving process demonstrated by ReAct is more factual and grounded, whereas CoT is more accurate in formulating reasoning structure but can easily suffer from hallucinated facts or thoughts.
We therefore propose to incorporate ReAct and CoT-SC , and let the model decide when to switch to the other method based on the following heuristics
: • ReAct → CoT-SC : when ReAct fails to return an answer within given steps, back off to CoT-SC .
We set 7 and 5 steps for HotpotQA and Fever respectively as we ﬁnd more steps will not improve ReAct performance.
• ReAct → CoT-SC : when the majority answer among n CoT-SC samples occurs less than n/ 2 times (i.e. internal knowledge might not support the task, thus reasoning samples are divergent), back off to ReAct .
Finetuning Due to the challenge of manually annotating reasoning traces and actions at scale, we consider a bootstraping approach similar to Zelikman et al.
(2022), using 3,000 trajectories with correct answers generated by ReAct (also for other baselines) to ﬁnetune smaller language models (PaLM-8/62B) to decode trajectories (all thoughts, actions, observations) conditioned on input questions/claims.
More details are in Appendix D.
3. R ESULTS AND O BSERVATIONS
ReAct outperforms Act consistently Table 1 shows HotpotQA and Fever results using PaLM540B as the base model with different prompting methods.
We note that ReAct is better than Act on both tasks, demonstrating the value of reasoning to guide acting, especially for synthesizing the ﬁnal answer, as shown in Figure 1 (1c-d).
Fine-tuning results 3 also conﬁrm
the beneﬁt of reasoning traces for more informed acting.
ReAct vs. CoT On the other hand, ReAct outperforms CoT on Fever (60.9 vs. 56.3) and slightly lags behind CoT on HotpotQA (27.4 vs. 29.4).
Fever claims for SUPPORTS/REFUTES might only differ by a slight amount (see Appendix B.1), so acting to retrieve accurate and up-to-date knowledge is vital.
To better understand the behavioral difference between ReAct and CoT on HotpotQA, we randomly sampled 50 trajectories with correct and incorrect answers (judged by EM) from ReAct and CoT respectively (thus 200 examples in total), and manually labeled their success and failure modes in
Table 2.
Some key observations are as follows:
• Hallucination is a serious problem for CoT , resulting in much higher false positive rate than ReAct (14% vs. 6%) in success mode, and make up its major failure mode (56%). [Table] [Table]
[Figure]
[Table] [Table] In contrast, the problem trajectory of ReAct is more grounded, fact-driven, and trustworthy, thanks to the access of an external knowledge base.
• While interleaving reasoning, action and observation steps improves ReAct ’s groundedness and trustworthiness, such a structural constraint also reduces its ﬂexibility in formulating reasoning steps, leading to more reasoning error rate than CoT .
we note that there is one frequent error pattern speciﬁc to ReAct , in which the model repetitively generates the previous thoughts and actions, and we categorize it as part of “reasoning error” as the model fails to reason about what the proper next action to take and jump out of the loop 2 .
• For ReAct , successfully retrieving informative knowledge via search is critical.
Noninformative search results, which counts for 23% of the error cases, derail the model thinking trajectory and give model a hard time to recover and reformulate thoughts.
This is perhaps an expected trade-off between factuality and ﬂexibility, which motivates our proposed strategies of combining two methods.
We provide examples for each success and failure modes in Appendix C.1.
ReAct + CoT -SC perform best for prompting LLMs Also shown in Table 1, the best prompting method on HotpotQA and Fever are ReAct → CoT-SC and CoT-SC → ReAct respectively.
Furthermore, Figure 2 shows how different methods perform with respect to the number of CoT-SC samples used.
While two ReAct + CoT-SC methods are advantageous at one task each, they both signiﬁcantly and consistently outperform CoT-SC across different number of samples, reaching
CoT-SC performance with 21 samples using merely 3-5 samples.
These results indicate the value of properly combining model internal knowledge and external knowledge for reasoning tasks.
ReAct performs best for ﬁne-tuning Figure 3 shows the scaling effect of prompting/ﬁnetuning four methods ( Standard , CoT , Act , ReAct ) on HotpotQA.
With PaLM-8/62B, prompting ReAct performs worst among four methods due to the difﬁculty to learn both reasoning and acting from in-context examples.
However, when ﬁnetuned with just 3,000 examples, ReAct becomes the best method among the four, with PaLM-8B ﬁnetuned ReAct outperforming all PaLM-62B prompting to the sub-optimal greedy decoding procedure, and future work using better deocoding (e.g.beam search) might help address this issue. [Figure] methods, and PaLM-62B ﬁnetuned ReAct outperforming all 540B prompting methods.
In contrast, ﬁnetuning Standard or CoT is signiﬁcantly worse than ﬁnetuning ReAct or Act for both PaLM8/62B, as the former essentially teaches models to memorize (potentially halluincated) knowledge facts, and the latter teaches models how to (reason and) act to access information from Wikipedia, a more generalizable skill for knowledge reasoning.
As all prompting methods are still signiﬁcantly far from domain-speciﬁc state-of-the-art approaches (Table 1), we believe ﬁnetuning with more human-written data might be a better way to unleash the power of ReAct .
D ECISION M AKING T ASKS
We also test ReAct on two language-based interactive decision-making tasks, ALFWorld and
WebShop, both of which feature complex environments that require agents to act over long horizons with sparse rewards, warranting the need for reasoning to act and explore effectively.
ALFWorld ALFWorld
(Shridhar et al., 2020b) (Figure 1(2)) is a synthetic text-based game designed to align with the embodied ALFRED benchmark (Shridhar et al., 2020a).
It includes 6 types of tasks in which an agent needs to achieve a high-level goal (e.g. examine paper under desklamp) by navigating and interacting with a simulated household via text actions (e.g. go to coffeetable 1, take paper 2, use desklamp 1).
A task instance can have more than 50 locations and take an expert policy more than 50 steps to solve, thus challenging an agent to plan and track subgoals, as well as explore systematically (e.g. check all desks one by one for desklamp).
In particular, one challenge built into
ALFWorld is the need to determine likely locations for common household items (e.g. desklamps will likely be on desks, shelfs, or dressers), making this environment a good ﬁt for LLMs to exploit their pretrained commonsense knowledge.
To prompt ReAct , we randomly annotate three trajectories from the training set for each task type, where each trajectory includes sparse thoughts that (1) decompose the goal, (2) track subgoal completion, (3) determine the next subgoal, and (4) reason via commonsense where to ﬁnd an object and what to do with it.
We show prompts used for ALFWorld in Appendix A.4.
Following Shridhar et al.
(2020b), we evaluate on 134 unseen evaluation games in a task-speciﬁc setup.
For robustness, we construct 6 prompts for each task type through each permutation of 2 annotated trajectories from the 3 we annotate.
Act prompts are constructed using the same trajectories, but without thoughts — since task instances are randomly chosen from the training set, it favors neither ReAct nor Act and provides a fair and controlled comparison to test the importance of sparse thoughts.
For baselines, we use BUTLER (Shridhar et al., 2020b), an imitation learning agent trained on 10 5 expert trajectories for each task type 3 .
WebShop Can ReAct also interact with noisy real-world language environments for practical applications?
We investigate WebShop (Yao et al., 2022), a recently proposed online shopping website environment with 1.18M real-world products and 12k human instructions.
Unlike ALFWorld,
Webshop contains a high variety of structured and unstructured texts (e.g. product titles, descriptions, and options crawled from Amazon), and requires an agent to purchase a product based on a user instruction (e.g. “I am looking for a nightstand with drawers. It should have a nickel ﬁnish, and [Table]
[Table] priced lower than $140”) through web interactions (e [Table] g. search “nightstand drawers”, choose buttons such as “color: modern-nickel-white” or “back to search”).
This task is evaluated by average score (percentage of desired attributes covered by the chosen product averaged across all episodes) and success rate (percentage of episodes where the chosen product satisﬁes all requirements) on 500 test instructions.
We formulate Act prompts with actions to search, choose product, choose options [Table] and buy, with ReAct prompts additionally reasoning to determine what to explore, when to buy [Table]
and what products options are relevant to the instruction.
See Table 5 for an example prompt [Table] and Tablefor model predictions in the Appendix.
We compare to an imitation learning (IL) method trained with 1,012 human annotated trajectories, and a imitation + reinforcement learning (IL + RL) method additionally trained with 10,587 training instructions.
Results ReAct outperforms Act on both ALFWorld
(Table 3) and Webshop (Table 4).
On
ALFWorld, the best ReAct trial achieves an average success rate of 71%, signiﬁcantly outperforming the best Act (45%) and BUTLER (37%) trials.
In fact, even the worse ReAct trial (48%) beats the best trial of both methods.
Moreover, the advantage of ReAct over Act is consistent across six controlled trials, with relative performance gain ranging from 33% to 90% and averaging 62%.
Qualitatively, we saw that, without any thoughts at all, Act fails to correctly decompose goals into smaller subgoals, or loses track of the current state of the environment.
Example trajectories comparing ReAct and Act can be found in Appendix B.2.1 and Appendix B.2.2.
On Webshop, one-shot Act prompting already performs on par with IL and IL+RL methods.
With additional sparse reasoning, ReAct achieves signiﬁcantly better performance, with an absolute 10% improvement over the previous best success rate.
By checking examples, we ﬁnd that ReAct is more likely to identify instruction-relevant products and options by reasoning to bridge the gap between noisy observations and actions (e.g. “For ‘space-saving ottoman bench for living room’, the item has options ‘39x18x18inch’ and ‘blue’ and seems good to buy.”).
However, existing methods are still far from the performance of expert humans (Table 4), who perform signiﬁcantly more product explorations and query re-formulations that are still challenging for prompting-based methods.
On the value of internal reasoning vs. external feedback To our knowledge, ReAct is the ﬁrst demonstration of combined reasoning and action using an LLM applied to an interactive environment within a closed-loop system.
Perhaps the closest prior work is Inner Monologue (IM), from Huang et al.
(2022b).
The motivation behind IM is very similar to that of ReAct , in which actions from an embodied agent are motivated by an eponymous “inner monologue”, i.e. internal thoughts and reasoning.
However, IM’s “inner monologue” is implemented as dense external feedback from an environment.
In particular, the thoughts experimented within an IM are limited to observations of the environment state and what needs to be completed by the agent for the goal to be satisﬁed.
In contrast, the thoughts described in ReAct are (1) free-form and unlimited, through which human annotation can induce speciﬁc policies for the LLM to follow, (2) abstract and diverse, including memory traces, strategization and error recovery, and (3) sparse, in that ReAct can decide when to think and avoid cluttering the LLM input history.
To demonstrate the differences between ReAct and IM, and to highlight the importance of internal reasoning vs. simple reactions to external feedback, we ran an ablation experiment using a thought pattern composed of IM-like dense external feedback.
Table 3 includes ReAct experiments on
[Table]
[Figure] ALFWorld with this IM-style prompting as an ablation experiment, labeled as ReAct-IM .
The same three expert trajectories used are reannotated with dense external feedback thoughts within these trajectories, that limit ReAct-IM to only think about (1) decomposing the current goal and (2) the current subgoal that needs to be completed.
In particular, ReAct-IM lacks thoughts that (1) determine when a subgoal is completed (2) determine what the next subgoal should be (3) inducing the LLM to refer to its internal pretraining knowledge to identify where items can be within the environment.
As can be seen in Table 3, ReAct substantially outperforms IM-style prompting ( vs. 53 overall success rate), with consistent advantages on ﬁve out of six tasks.
Qualitatively, we observed that ReAct-IM often made mistakes in identifying when subgoals were ﬁnished, or what the next subgoal should be, due to a lack of high-level goal decomposition.
Additionally, many
ReAct-IM trajectories struggled to determine where an item would likely be within the ALFWorld environment, due to a lack of commonsense reasoning.
Both shortcomings can be addressed in the ReAct paradigm.
An example prompt for ReAct-IM can be found in Appendix A.4, and an example trajectory in Appendix B.2.3.
Example of human-in-the-loop behavior correction Finally, we also explore human-in-the-loop interaction with ReAct , to allow a human to inspect and edit ReAct ’s reasoning traces. Figureshows that by simply removing a hallucinating sentence in Act 17 and adding some hints in Act 23, ReAct can be made to change its behavior drastically to align with these human thought edits and succeed in the task. From a human perspective, solving such a task becomes signiﬁcantly easier, from typing tens of actions to only editing a couple of thoughts, which enables new forms of human-machine collaboration. We note that such a policy edit on-the-go is difﬁcult for Act and previous RL methods, as a human cannot change the model parameters, and changing a few actions might not edit the rest of the model behavior.
This paradigm is also more than human dialogue to update the goal or subgoal as in Huang et al. (2022b) — while editing ReAct thoughts can do these, it can also modify the model’s internal belief, reasoning styles, or anything the ﬂexible thought space supports, for better task solving. We believe this is an exciting direction for human alignment and leave more systematic study as future work.  R ELATED W ORK
Language model for reasoning Perhaps the most well-known work of using LLMs for reasoning is Chain-of-Thought (CoT) (Wei et al., 2022), which reveals the ability of LLMs to formulate their own “thinking procedure” for problem solving.
Several follow-up works have since been performed, including least-to-most prompting for solving complicated tasks (Zhou et al., 2022), zero-shotCoT (Kojima et al., 2022), and reasoning with self-consistency (Wang et al., 2022a).
Recently, (Madaan & Yazdanbakhsh, 2022) systematically studied the formulation and structure of CoT, and observed that the presence of symbols, patterns and texts is crucial to the effectiveness of CoT. Other work has also been extended to more sophisticated reasoning architecture beyond simple prompting.
For example Selection-Inference (Creswell et al., 2022) divides the reasoning process into two steps of “selection” and “inference”.
STaR (Zelikman et al., 2022) bootstraps the reasoning process by ﬁnetuning the model on correct rationales generated by the model itself.
Faithful reasoning (Creswell & Shanahan, 2022) decomposes multi-step reasoning into three steps, each performed by a dedicated
LM respectively.
Similar approaches like Scratchpad (Nye et al., 2021), which ﬁnetunes a LM on intermediate computation steps, also demonstrate improvement on multi-step computation problems.
In contrast to these methods, ReAct performs more than just isolated, ﬁxed reasoning, and integrates model actions and their corresponding observations into a coherent stream of inputs for the model to reason more accurately and tackle tasks beyond reasoning (e.g. interactive decision making).
Language model for decision making The strong capability of LLMs has enabled them to perform tasks beyond language generation, and it is becoming more popular to take advantage of LLMs as a policy model for decision making, especially in interactive environments.
WebGPT (Nakano et al., 2021) uses an LM to interact with web browsers, navigate through web pages, and infer answers to complicated questions from ELI5 (Fan et al., 2019).
In comparison to ReAct , WebGPT does not explicitly model the thinking and reasoning procedure, instead rely on expensive human feedback for reinforcement learning.
In the conversation modeling space, chatbots like BlenderBot (Shuster et al., 2022) and Sparrow (Glaese et al., 2022) also train LMs to make decision about search API calls.
Unlike ReAct , they do not explicitly consider the reasoning procedure either, and also relies on expensive datasets and human feedback collections for policy learning.
In contrast, ReAct learns a policy in a much cheaper way, since the decision making process only requires language description of the reasoning procedure.
4 LLMS have also been increasingly employed in interactive and embodied environments for planning and decision making.
Perhaps most relevant to ReAct in this respect are SayCan (Ahn et al., 2022) and Inner Monologue (Huang et al., 2022b), which use LLMs for robotic action planning and decision making.
In SayCan, LLMs were prompted to directly predict possible actions a robot can take, which is then reranked by an affordance model grounded on the visual environments for ﬁnal prediction.
Inner Monologue made further improvements by adding the eponymous “inner monologue", which is implemented as injected feedback from the environment.
To our knowledge, Inner Monologue is the ﬁrst work that demonstrates such a closed-loop system, which ReAct builds on.
However, we argue that Inner Monologue does not truly comprise of inner thoughts — this is elaborated in Section 4.
We also note that leveraging language as semantically-rich inputs in the process of interactive decision making has been shown to be successful under other settings (Abramson et al., 2020; Karamcheti et al., 2021; Huang et al., 2022a; Li et al., 2022).
It is becoming more evident that with the help of LLMs, language as a fundamental cognitive mechanism will play a critical role in interaction and decision making.
What is more, progress in LLMs has also inspired the development of versatile and generalist agents like Reed et al.
C ONCLUSION
We have proposed ReAct – a simple yet effective method for synergizing reasoning and acting in large language models.
Through a diverse set of experiments on multi-hop question-answering, fact checking, and interactive decision-making tasks, we show that ReAct leads to superior performance with interpretable decision traces.
Despite the simplicity of our method, complex tasks with large action spaces require more demonstrations to learn well, which unfortunately can easily go beyond the input length limit of in-context learning.
We explore the ﬁne-tuning approach on HotpotQA with initial promising results, but learning from more high-quality human annotations will be the desiderata to further improve the performance.
Scaling up ReAct with multi-task training and combining it with complementary paradigms like reinforcement learning could result in stronger agents that further unlock the potential of LLMs for more applications.
A CKNOWLEDGMENTS
We thank the support and feedback of many people from Google Brain team and Princeton NLP
Group.
This work was supported in part by the National Science Foundation under Grant No.
Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of the National Science Foundation.
ActionSearch[Stranger Things] Observation
Stranger Things is an American science fiction horror drama television series created by the Duffer Brothers.
Set in the 1980s, primarily in the fictional town of Hawkins, Indiana, the series centers on a number of mysteries and supernatural events occurring around the town and their impact on an ensemble of child and adult characters.
Action
Finish[REFUTES]
Claim
Beautiful reached number two on the Billboard Hot 100 in 2003.
Action
Search[Beautiful] Observation
Could not find [Beautiful].
Similar: [’Beautiful’, ’Beautiful, Beautiful’, ’A Beautiful Mind (film)’, ’Beautiful (Christina Aguilera song)’, ’Life Is
Beautiful’].
Action
Search[Beautiful (Christina Aguilera song) Observation"Beautiful" is a song recorded by American singer Christina Aguilera for her fourth studio album, Stripped (2002).
Action
Lookup[Billboard Hot 100] Observation(Result 1 / 3)
The song peaked at number two on the Billboard Hot 100 in the United States, where it was certified Gold for 500,000 units shipped.
Action Finish[NOT ENOUGH INFO] CoT
Determine if there is Observation that SUPPORTS or REFUTES a Claim, or if there is NOT ENOUGH INFORMATION.
Claim
Nikolaj Coster-Waldau worked with the Fox Broadcasting Company.
Thought
Nikolaj William Coster-Waldau appeared in the 2009 Fox television film Virtuality, so he has worked with the Fox Broadcasting Company. Answer SUPPORTS
Claim
Stranger Things is set in Bloomington, Indiana.
Thought
Stranger Things is in the fictional town of Hawkins, Indiana, not in Bloomington, Indiana.
Answer
REFUTES
Continued on next page
[Table]
– Continued
[Table]
page
[Table] Beautiful [Table] two
[Table]
in [Table] [Table] [Table]
[Table]
Determine if there is Observation that SUPPORTS or REFUTES a Claim, or if there is NOT ENOUGH
[Table]
[Table]
Nikolaj Coster-Waldau worked with the Fox Broadcasting Company.
[Table]
I need to search Nikolaj Coster-Waldau and find if he has worked with the Fox Broadcasting Company.
[Table]
[Nikolaj Coster-Waldau]
[Table]
Nikolaj William Coster-Waldau (born 27 July 1970) is a Danish actor and producer.
He graduated from the Danish National School of Performing Arts in Copenhagen in 1993,[1] and had his breakthrough role in Denmark with the film Nightwatch (1994).
He played Jaime Lannister in the HBO fantasy drama series Game of Thrones, for which he received two Primetime Emmy Award nominations for Outstanding Supporting Actor in a Drama Series..
Coster-Waldau has appeared in numerous films in his native Denmark and Scandinavia, including Headhunters (2011) and A Thousand Times Good Night (2013).
In the U.S, his debut film role was in the war film Black Hawk Down (2001), playing Medal of Honor recipient Gary Gordon.[2]
He then played a detective in the short-lived Fox television series New Amsterdam (2008), and appeared in the 2009 Fox television film Virtuality, originally intended as a pilot.
Thought
Because he "appeared in the 2009 Fox television film Virtuality", he should have worked with the Fox Broadcasting Company.
[Table]
Stranger Things is set in Bloomington, Indiana.
Thought
[Table]
I should search for Stranger Things, and see if it is set in Bloomington, Indiana.
[Table]
[Table] Observation
Stranger Things is an American science fiction horror drama television series created by the Duffer Brothers.
Set in the 1980s, primarily in the fictional town of Hawkins, Indiana, the series centers on a number of mysteries and supernatural events occurring around the town and their impact on an ensemble of child and adult characters.
Thought
The observation says that it is set in a "fictional town of Hawkins, Indiana", so it is not set in Bloomington.
[Table]
Finish[REFUTES]
Claim
Beautiful reached number two on the Billboard Hot 100 in 2003.
Thought
I need to search the song Beautiful and find if it reached number two on the Billboard Hot 100 in 2003.
[Table]
Search[Beautiful] Observation
Could not find [Beautiful].
Similar: [’Beautiful’, ’Beautiful, Beautiful’, ’A Beautiful Mind (film)’, ’Beautiful (Christina Aguilera song)’, ’Life Is
Beautiful’].
Thought
From suggestions, I should search "Beautiful (Christina Aguilera song)" to find the song.
[Table]
Search[Beautiful (Christina Aguilera song)]
[Table]
"Beautiful" is a song recorded by American singer Christina Aguilera for her fourth studio album, Stripped (2002).
Thought
It does not mention Billboard, so I need to look up "Billboard Hot 100" to find if it reached number two on it in 2003.
[Table]
Lookup
[Billboard Hot 100]
[Table]
(Result 1 / 3)
The song peaked at number two on the Billboard Hot 100 in the United States, where it was certified Gold for 500,000 units shipped.
[Table]
It only says the song peaked at number two on the Billboard Hot 100, but not if it was in 2003.
I am not sure if this claim is true or not.
[Table]
Finish[NOT ENOUGH INFO]
A. W EBSHOP
[Table]
[Table] [Table]
[Table]
B T RAJECTORIES B. FEVER T RAJECTORIES
We provide some trajectories produced by ReAct , CoT and Act on several randomly selected examples from the FEVER dev set below.
Search results are omitted to space
[Table]
Example 2491 (gt: REFUTES)
ReActClaim:
Bermuda Triangle is a loosely-defined region in the Pacific Ocean.
Thought 1:
I need to search Bermuda Triangle and find if it is in the Pacific Ocean.
Action 1:
Search[Bermuda Triangle] Observation 1:
Reportedly haunted locations...
Thought 2:
The observation says that it is in the western part of the North Atlantic Ocean, so it is not in the Pacific Ocean.
Action 2:
Finish[REFUTES] Observation 2:
Episode finished
ActClaim:
Bermuda Triangle is a loosely-defined region in the Pacific Ocean.
Action: Search[Bermuda Triangle] Observation: Reportedly haunted locations:... Action 2:
Finish[REFUTES] Observation 2:
Episode finished
CoTClaim:
Bermuda Triangle is a loosely-defined region in the Pacific Ocean.
Thought:
Let’s think step by step.
The Bermuda Triangle is a loosely-defined region in the Atlantic Ocean.
The Pacific Ocean is not the Atlantic Ocean.
Answer:
REFUTES
Example 5908 (gt: SUPPORTS)
ReActClaim:
Princess Mononoke is a film.
Thought 1:
I need to search Princess Mononoke and find if it is a film.
Action 1:
Search[Princess Mononoke] Observation 1:
Princess Mononoke ... Thought 2:
From the observation, it says that Princess Mononoke is a film.
Action 2:
Finish[SUPPORTS] Observation 2:
Episode finished
ActClaim:
Princess Mononoke is a film.
Action 1:
Search[Princess Mononoke] Observation 1:
Princess Mononoke ... Action 2:
Finish[SUPPORTS] Observation 2:
Episode finished
CoTClaim:
Princess Mononoke is a film.
Thought:
Let’s think step by step.
Princess Mononoke is a movie, and movies are films.
Answer:
SUPPORTS
Example 1951 (gt: REFUTES) ReAct
Claim:
Soyuz was part of the American space program.
Thought 1:
I need to search Soyuz and find if it was part of the American space program.
Action 1:
Search[Soyuz] Observation 1:
Soyuz is a transliteration...
Thought 2:
It does not mention the American space program, so I need to search "American space program" to find if it was part of it.
Action 2:
Search[American space program] Observation 2:
The space policy... Thought 3:
It does not mention Soyuz, so I am not sure if this claim is true or not.
Action 3:
Finish[NOT ENOUGH INFO] Observation 3:
Episode finished
ActClaim:
Soyuz was part of the American space program. Action 1: Search[Soyuz] Observation 1: Soyuz is a transliteration...
Action 2: Finish[NOT ENOUGH INFO] Observation 2: Episode finished
CoTClaim:
Soyuz was part of the American space program.
Thought:
Let’s think step by step.
Soyuz is a Russian spacecraft.
The American space program is NASA.
NASA and Russia have worked together on the
International Space Station.
Answer:
SUPPORTS
Example 3208 (gt: REFUTES)
ReActClaim:
Reign Over Me is an American film made in 2010.
Thought 1:
I need to search for Reign Over Me and find if it is an American film made in 2010.
Action 1:
Search[Reign Over Me] Observation 1:
Reign Over Me ...
Thought 2:
The observation says that it is an American film made in 2007, so it is not made in 2010.
Action 2:
Finish[REFUTES] Observation 2:
Episode finished
ActClaim:
Reign Over Me is an American film made in 2010.
Action 1:
Search[Reign Over Me] Action 2:
Finish[SUPPORTS] Observation 2:
Episode finished
CoTClaim:
Reign Over Me is an American film made in 2010.
Thought:
Let’s think step by step.
First, is Reign Over Me an American film?
Yes, it is.
Second, was it made in 2010?
Yes, it was.
Answer:
SUPPORTS
B. ALFW ORLD T RAJECTORIES
We provide an example trajectory of ReAct , Act , and ReAct-IM on the same ALFWorld game, pick_clean_then_place_in_recep-Knife-None-CounterTop-10/ trial_T20190909_110445_675754 .
B.2. R E A CT TRAJECTORY
ReAct is able to correctly ﬁnd the knife, clean it, and then place it on a countertop.
[Table]
The cabinet 6 is closed.
> go to drawerThe drawer 1 is closed.
> go to drawerThe drawer 2 is closed.
> go to drawerThe drawer 3 is closed.
> go to countertopOn
[Table]
to
[Table]
[Table]
B.2. A CT TRAJECTORY
Act is able to correctly ﬁnd the knife
[Table]
but tries to clean it using a sinkbasin before actually going to the sinkbasin.
From there on, the model is unable to recover
[Table]
and is stuck repeating the same sequence of commands.
With ReAct , an existing thought would have concretely identiﬁed that the knife had been taken
[Table]
and that the next subgoal was to go to a sinkbasin to clean it
[Table] [Table]
B.2. R E A CT -IM TRAJECTORY
ReAct-IM is able to correctly ﬁnd the knife, but is unable to clean it as well.
The LLM then gets stuck trying to place this knife on a countertop.
One possible issue is the incorrect thought think:
I need to find a clean knife. , which seems to trick the LLM in believing that the knife is already cleaned.
In ReAct , this sequence of subgoals would be concretely captured by each thought.
[Table]
B. W EBSHOP T RAJECTORIES
[Table]
C M ORE A NALYSIS C. S UCCESS AND F AILURE M ODES A NALYSIS
We provide some examples corresponding to the success and error mode analysis given in Sec. 3.3.
Search results and non-representative steps are omitted to reduce space.
Success: True positive
ReActQuestion:
Author David Chanoff has collaborated with a U.S. Navy admiral who served as the ambassador to the United Kingdom under which President?
Thought 1:
I need to search David Chanoff and find the U.S. Navy admiral he collaborated with.
Action 1:
Search[David Chanoff] Observation 1:
David Chanoff is a noted author... Action 2:
Search[William J. Crowe] Observation 2:
William J. Crowe is... Thought 3:
William J. Crowe served as the ambassador to the United Kingdom under President Bill Clinton.
So the answer is Bill Clinton.
Action 3:
Finish[Bill Clinton] Observation 3:
Episode finished
CoTQuestion:
What was the name of the team that represented an Arizona university in the conference that covers a broad expanse of the western United States, including Arizona, California, and New Mexico?
Thought:
Let’s think step by step.
The name of the team that represented an Arizona university in the conference that covers a broad expanse of the western
United States, including Arizona, California, and New Mexico is Arizona State Sun Devils.
Answer:
Arizona State Sun Devils
Success: False positive
ReActQuestion:
Did the board game San Marco or About Time come out first?
Thought 1:
I need to search San Marco and About Time, and find which came out first.
Action 1:
Search[San Marco] Observation 1:
San Marco is one of the ...
The was built in venice.
Thought 2:
San Marco is a sestiere of Venice.
I need to search About Time next.
Action 2:
Search[About Time] Observation 2:
Could not find [About Time].
Similar: [...].
Thought 3:
About Time is a 2013 film.
So San Marco came out first.
Action 3:
Finish[San Marco] Observation 3:
Episode finished
CoTQuestion:
Was Vanderbilt University or Emory University founded first?
Thought:
Let’s think step by step.
Vanderbilt University was founded in 1873.
Emory University was founded in 1836.
1873 (Vanderbilt University) > 1836 (Emory
University), so Vanderbilt University was founded first.
Answer:
Emory University
Failure: Reasoning error
ReActQuestion:
What was the middle name of the actress who was cast alongside Harvey Fierstein, Linda Hart, Dick Latessa, Clarke Thorell, Mary Bond Davis, Laura
Bell Bundy, Matthew Morrison, Corey Reynolds, and Marissa Jaret Winokur in Hairspray?
Thought 1:
I need to search Harvey Fierstein, Linda Hart, Dick Latessa, Clarke Thorell, Mary Bond Davis, Laura Bell Bundy, Matthew Morrison, Corey Reynolds, and
Marissa Jaret Winokur, find the actress that was cast alongside them in Hairspray, then find its middle name.
Action 1:
Search[Harvey Fierstein] Observation 1:
Harvey Forbes Fierstein ... Thought 2:
Harvey Fierstein is an actor, playwright, and screenwriter.
I need to search Linda Hart next...
Action 2:
Search[Linda Hart]...
CoTQuestion:
Who is older Danny Green or James Worthy?
Thought:
Let’s think step by step.
Danny Green was born in 1987.
James Worthy was born in 1961.
1987 (Danny Green) > 1961 (James Worthy), so Danny Green is older.
Answer:
Danny Green
Failure: Search error
ReActQuestion:
What does the goddess associated with the goddess frigg consists of what tales?
Thought 1:
I need to search goddess frigg and find its associated goddess.
Action 1:
Search[goddess frigg]
Observation 1:
Could not find goddess frigg.
Similar: [’Frigg’, ’Frigg and Freyja common origin hypothesis’, ’Freyja (section Relation to Frigg and other goddesses and figures)’, ’Eir (redirect from Eir (goddess))’, ’Baldr’].
Thought 2:
I need to search Frigg instead...
Failure: Hallucination
CoTQuestion:
What was the first year a scientific journal published by an organization located in the Bronx was published?
Thought:
Let’s think step by step.
The first year a scientific journal published by an organization located in the Bronx was published was 1916.
Answer: 1916 (Label: 1909)
Failure: Label ambiguity ReAct
Question:
What nationality is the man who developed the dictator game?
Thought 1:
I need to search dictator game, find the man who developed it, then find his nationality.
Action 1:
Search[dictator game] Observation 1:
The dictator game is a popular experimental instrument in social psychology and economics...
Action 3:
Search[Amos Tversky] Observation 3:
Amos Nathan Tversky...
Thought 4:
Amos Tversky is an Israeli cognitive and mathematical psychologist.
So the man who developed the dictator game is Israeli.
Action 4:
Finish[Israeli] (label: Israel-American)
CoTQuestion:
Kurt Fuller from Stockton, CA appeared in which American Detective Comedy Drama television series created by Steve Franks?
Thought:
Let’s think step by step.
Kurt Fuller from Stockton, CA appeared in Psych.
Psych is an American Detective Comedy Drama television series created by
Steve Franks.
Answer:
Psych (label: Psych is an American detective comedy-drama)
D H OTPOT QA F INETUNING D ETAILS
For all ﬁnetuning we use a batch size of 64.
On PaLM-8B, we ﬁnetune ReAct and Act methods for 4 , 000 steps and Standard and CoT methods for 2 , 000 steps.
On PaLM-62B, we ﬁnetune
ReAct and Act methods for 4 , 000 steps and Standard and CoT methods for 1 , 000 steps.
We ﬁnd ReAct and Act methods generally beneﬁt from more training steps (and more training data), while Standard and CoT methods degrade soon after ﬁnetuning.