Paper: Prospector: Improving LLM Agents with Self-Asking and Trajectory Ranking (ICLR-2024)

Reviewer comment (Experiments):
It’s not clear how much of the overall performance improvement is just due to giving the LLM multiple attempts at a single question with the trajectory ranking process. Further experiments disentangling this would be helpful: for example, if we used the same LLM critics and trajectory ranking process with the ReAct prompt, would it perform on par with Prospector (these experiments seem to be present for ALFWorld but not WebShop)? Would majority voting at every step, which also allows multiple trajectory attempts but without an explicit LLM critic, be less useful than using the LLM-based critic as in Prospector?

Author response:
**WA1.** To address this important suggestion, we conducted **additional experiments on WebShop**. To provide a more clear view on the performance of each component of Prospector, we conducted experiments with four different settings: (1) ReAct only, (2) AskAct only, (3) ReAct + Trajectory Ranking (TR), and (4) AskAct + TR. To expedite experiments and reduce costs, we used open-source LLMs (i.e., Llama-2-70B for the LLM Actor and FLAN-T5-3B for the LLM Critic) instead of closed-source LLMs. The additional experiment results are summarized in the bottom section of the following table.

Some findings from the additional experiments can be summarized as follows:
- On the WebShop environment, **Llama-2-70B**, one of representative open-source LLMs **can achieve comparable performance with text-davinci-002**, one of the most powerful LLMs.
- In both cases of text-davinci-002 and Llama-2-70B, **AskAct meaningfully improves the success rate compared to ReAct:** from 35.8 to 39.8 on text-davinci-002, and **from 37.6 to 42.2 on Llama-2-70B**. This means that AskAct, a simple prompting method that adds extra question prompts on ReAct, can be effective.
- **ReAct + TR (the setting requested by the reviewer) can improve ReAct from 37.6 to 42.2** in the success rate. However, ReAct + TR is less efficient in terms of the computation complexity, since **it requires 8x larger computation than AskAct** to achieve the similar performance.
- **AskAct + TR further improves the success rate of AskAct (from 42.2 to 43.6)**, and provides better performance than ReAct + TR (42.2).

We will add the results in the revised paper, and upload it during the author response period.

| Method | LLM Actor | LLM Critic | Reward | Success Rate |
|---|---|---|:---:|:---:|
| Human (expert) | - | - | 82.1 | 59.6 |
| Human (average) | - | - | 75.5 | 50.0 |
| IL | - | - | 59.9 | 29.1 |
| IL + RL | - | - | 62.4 | 28.7 |
| ReAct | PaLM-540B | - | 66.6 | 40.0 |
| ReAct | text-davinci-002 | - | 63.3 | 35.8 |
| ReAct + Reflexion (k=8) | text-davinci-002 | - | - | 35.0 |
| AskAct (Prospector) | text-davinci-002 | - | 66.5 | 39.8 |
| **AskAct + TR (k=8) (Prospector)** | text-davinci-002 | text-davinci-002 | 69.3 | **41.4** |
| ReAct | Llama-2-70B | - | 62.3 | 37.6 |
| AskAct (Prospector) | Llama-2-70B | - | 68.6 | 42.2 |
| ReAct + TR (k=8) (Prospector) | Llama-2-70B | FLAN-T5-3B (SFT) | 69.3 | 42.2 |
| **AskAct + TR (k=8) (Prospector)** | Llama-2-70B | FLAN-T5-3B (SFT) | 70.2 | **43.6** |

Reviewer comment (Novelty):
The AskAct process does not seem like a novel contribution in and of itself, as it was proposed in Measuring and Narrowing the Compositionality Gap in Language Models (Press et al., 2022). While the authors note that the Self-Ask work was developed for QA tasks specifically, applying the same general technique of prompting the LLM to ask itself a limited set of questions to reason is a limited contribution. In particular, it seems like AskAct was not applied to the ALFWorld benchmark for the Prospector agent, and only tested in WebShop as a single fixed question that asks "which observed object is most proper to select" (shown in Figure 2 and Table 13 and 16). Further experiments eliciting different types of self-asked questions across all the tasks would strengthen this contribution.

Author response:
**WA2.** To reflect this important suggestion, we conducted **additional experiments on ALFWorld**. Similar to the addtional experiments on WebShop, we conducted experiments with four different settings: (1) ReAct only, (2) AskAct only, (3) ReAct + Trajectory Ranking (TR), and (4) AskAct + TR.

Some findings from the additional experiments can be summarized as follows:
- **AskAct effectively improves the success rate of ReAct (from 41.0 to 56.7)**. Note that a LLM with lower temperature provides slightly better performance in case of single sampling.
- Since AskAct provides a better baseline, **AskAct + TR can achieve much better performance with less sampling** (e.g., AskAct only (56.7) comparable with ReAct + TR (k=2) (56.0)).
- We emphasize that **AskAct and TR can make an effective synergy in improving LLM agents in terms of both performance and efficiency**.

We will add the results and the AskAct prompt for ALFWorld in the revised paper.

| Method | LLM Actor | LLM Critic | k=1 | 2 | 3 | 4 | 5 |
| --- | --- | --- | :---: | :---: | :---: | :---: | :---: |
| BUTLER (RL) | -| - | 37.0 | - | - | - | - |
| Act | PaLM-540B | - | 45.0 | - | - | - | - |
| ReAct | PaLM-540B | - | 70.9 | - | - | - | - |
| ReAct | text-davinci-002 (temp=0.0) | - | 78.4 | - | - | - | - |
| ReAct + Reflexion | text-davinci-002 | - | - | - | - | - | 86.0 |
| **ReAct + TR (Prospector)** | text-davinci-002 (temp=0.8) | text-davinci-002 | 71.6 | - | 93.3 | - | **95.5** |
| ReAct | Llama-2-70B (temp=0.0) | - | **41.0** | - | - | - | - |
| AskAct (Prospector) | Llama-2-70B (temp=0.0) | - | **56.7** | - | - | - | - |
| ReAct + TR (Prospector) | Llama-2-70B (temp=0.8) | FLAN-T5-3B (SFT) | 33.6 | 59.0 | 69.4 | 73.1 | **77. 6** |
| **AskAct + TR (Prospector)** | Llama-2-70B (temp=0.8) | FLAN-T5-3B (SFT) | 53.7 | 76.1 | 80.6 | 84.3 | **86.6** |

Reviewer comment (Experiments):
It would be helpful to have an ablation with the trajectory ranking only, and no intermediate self-asking step for WebShop, as is shown in ALFWorld. Is most of the juice coming from the critic-based trajectory ranking?

Author response:
**A1.** This question is closely related to the weakness 1. Please see our response to the weakness 1 above.

Reviewer comment (Evaluation):
Limited discussion on the limitations of the reward prediction models used for Trajectory Ranking.

Author response:
We use a LLM as a reward model (named LLM Critic) that predicts the expected reward of a trajectory generated by the LLM Actor. **In this paper, we consider two types of LLM Critics: (1) few-shot LLM Critic and (2) fine-tuned LLM Critic.** A few-shot LLM Critic takes few examples of (trajectory, reward) as inputs, and generates a response as a reward. A fine-tuned LLM Critic is fine-tuned on a dataset of (trajectory, reward), and generate response as the reward for a given input trajectory. A few-shot LLM Critic has some advantages of generality, since it does not need to be trained on a specific environment. But, it may provide low prediction accuracy, as shown in the case of the WebShop environment. In contrast, a fine-tuned LLM Critic can provide high reward prediction accuracy, since it is trained on trajectories generated from an environment. Recently, building a general reward model is investigated as an important research problem. One of the representative work is Prometheus [1]. **We believe that Prospector can benefit from the general and accurate reward models like Prometheus.**  

[1] Seungone Kim, et al., "Prometheus: Inducing Fine-grained Evaluation Capability in Language Models", arXiv:2310.08491.

Reviewer comment (Evaluation):
How does the computational complexity of Prospector compare to that of existing methods like ReAct and Reflexion?

Author response:
We present an analysis of the complexity of each method.  
Let $N$ be the number of tokens involved in a single trajectory and $m$ the number of generated trajectories. Assuming a transformer model is used as the backbone, the complexity of ReAct is $O(N^2)$.  
Reflexion generates a sequence of $m$ trajectories. Assuming all past trajectories are given to the model, Reflexion has a complexity of $O(m^2N^2)$. Note that this may vary depending on how the method is implemented.  
Our method involves sampling $m$ independent trajectories and ranking them. This process has a complexity of $O(mN^2)$. **Although ReAct is more efficient than our approach, our method performs much better.** **Our method is further more efficient and performs better than Reflexion based on the analysis above.**

Reviewer comment (Evaluation):
Could you elaborate on the reward prediction models used in Trajectory Ranking? What are the limitations of the reward prediction models you used, and how do they impact the overall performance of Prospector?

Author response:
We use a LLM as a reward model (named LLM Critic) that predicts the expected reward of a trajectory generated by the LLM Actor. **In this paper, we consider two types of LLM Critics: (1) few-shot LLM Critic and (2) fine-tuned LLM Critic.** A few-shot LLM Critic takes few examples of (trajectory, reward) as inputs, and generates a response as a reward. A fine-tuned LLM Critic is fine-tuned on a dataset of (trajectory, reward), and generate response as the reward for a given input trajectory. A few-shot LLM Critic has some advantages of generality, since it does not need to be trained on a specific environment. But, it may provide low prediction accuracy, as shown in the case of the WebShop environment. In contrast, a fine-tuned LLM Critic can provide high reward prediction accuracy, since it is trained on trajectories generated from an environment. Recently, building a general reward model is investigated as an important research problem. One of the representative work is Prometheus [1]. **We believe that Prospector can benefit from the general and accurate reward models like Prometheus.**  

[1] Seungone Kim, et al., "Prometheus: Inducing Fine-grained Evaluation Capability in Language Models", arXiv:2310.08491.

Reviewer comment (Evaluation):
Are there specific types of questions or domains where the Self-Asking mechanism is more or less effective?

Author response:
In this study, **we find that Self-Asking is considerably effective in both ALFWorld and WebShop.** To answer this thoughtful question, we conducted additional experiments on ALFWorld. As shown in the following table, **AskAct effectively improves the success rate of ReAct (from 41.0 to 56.7).**  

| Method | LLM Actor | LLM Critic | Success Rate (%) |  
| --- | --- | --- | --- |  
| ReAct | Llama-2-70B | - | **41.0** |  
| ReAct + TR (k=5) (Prospector) | Llama-2-70B | FLAN-T5-3B (SFT) | 77.6 |  
| AskAct | Llama-2-70B | - | **56.7** |  
| AskAct + TR (k=5) (Prospector) | Llama-2-70B | FLAN-T5-3B (SFT) | 86.6 |  

Self-Asking is a prompting method based on a ReAct prompt, and **it simply adds few self-asking steps before reasoning steps**. In the following example AskAct prompt for ALFWorld, we highlight the added self-asking steps. **In ALFWorld, the self-asking step checks if a target object is found.** This self-asking step significantly reduces missing a target object in a place (due to hallucination). Further, it can elicit a proper next action, if the target object is found.  

---  

You are in the middle of a room. Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1.  

Your task is to: put some spraybottle on toilet.  

\> think: To solve the task, I need to find and take a sparybottle, then put it on toilet.  

OK.  

\> think: First I need to find a spraybottle. A spraybottle is more likely to appear in cabinet (1-4), countertop (1), toilet (1), sinkbasin (1-2), garbagecan (1). I can check one by one, starting with cabinet 1.  

OK.  

\> go to cabinet 1  

On the cabinet 1, you see a cloth 1, a soapbar 1, a soapbottle 1.  

\> **think: Did I find a spraybottle?**  

OK.  

\> **think: No, I did not. I need to go other place to find a spraybottle.**  

OK.  

\> go to cabinet 2  

The cabinet 2 is closed.  

\> open cabinet 2  

You open the cabinet 2. The cabinet 2 is open. In it, you see a candle 1, and a spraybottle 2.  

\> **think: Did I find a spraybottle?**  

OK.  

\> **think: Now I find a spraybottle (2). Next, I need to take it.**  

OK.  

\> take spraybottle 2 from cabinet 2

Reviewer comment (Experiments):
Unclear why the authors do not show prospector with askact + trajectory ranking results on Alfworld (Table 2,4). I’d encourage the authors to do this for the sake of completeness. Likewise, it would be good to see prospector with react + trajectory ranking on webshop (table 7).

Author response:
To answer this important question, we conducted **additional experiments on ALFWorld**. We conducted experiments with four different settings: (1) ReAct only, (2) AskAct only, (3) ReAct + Trajectory Ranking (TR), and (4) AskAct + TR. To expedite experiments and reduce costs, we used open-source LLMs (i.e., Llama-2-70B for the LLM Actor and FLAN-T5-3B for the LLM Critic) instead of closed-source LLMs. The additional experiment results are summarized in the bottom section of the following table.

Some findings from the additional experiments can be summarized as follows:
- **AskAct effectively improves the success rate of ReAct (from 41.0 to 56.7)**. Note that a LLM with lower temperature provides slightly better performance in case of single sampling.
- **Since AskAct provides a better baseline, AskAct + TR can achieve much better performance with less sampling** (e.g., AskAct only (56.7) comparable with ReAct + TR (k=2) (56.0)).
- We emphasize that **AskAct and TR can make an effective synergy in improving LLM agents in terms of both performance and efficiency**.

| Method | LLM Actor | LLM Critic | k=1 | 2 | 3 | 4 | 5 |
| --- | --- | --- | :---: | :---: | :---: | :---: | :---: |
| ReAct | text-davinci-002 (temp=0.0) | - | 78.4 | - | - | - | - |
| ReAct + Reflexion | text-davinci-002 | - | - | - | - | - | 86.0 |
| **ReAct + TR (Prospector)** | text-davinci-002 (temp=0.8) | text-davinci-002 | 71.6 | - | 93.3 | - | **95.5** |
| ReAct | Llama-2-70B (temp=0.0) | - | **41.0** | - | - | - | - |
| AskAct (Prospector) | Llama-2-70B (temp=0.0) | - | **56.7** | - | - | - | - |
| ReAct + TR (Prospector) | Llama-2-70B (temp=0.8) | FLAN-T5-3B (SFT) | 33.6 | 59.0 | 69.4 | 73.1 | **77. 6** |
| **AskAct + TR (Prospector)** | Llama-2-70B (temp=0.8) | FLAN-T5-3B (SFT) | 53.7 | 76.1 | 80.6 | 84.3 | **86.6** |

Reviewer comment (Evaluation):
Table 5,8: Why is few-shot reward prediction accuracy of LLM critic lower with more shots (3-shot vs. 2-shot)?

Author response:
This is due to the **limited context length** of text-davinci-002 (i.e., 4096). In both ALFWorld and WebShop, 3-shot examples often exceed the context length, and the last example is truncated.

Reviewer comment (Evaluation):
It seems that Prospector would be slower than React or reflexion because of additional reasoning that it does using more LLM calls. For real world interactive decision making tasks, it might be useful for the authors to also report compute time needed to decide the next action during the task execution. To that end, it would be great to also add a limitation section.

Author response:
Let $N$ be the number of tokens involved in a single trajectory and $m$ the number of generated trajectories. Assuming a transformer model is used as the backbone, the complexity of ReAct is $O(N^2)$. Reflexion generates a sequence of $m$ trajectories. Assuming all past trajectories are given to the model, Reflexion has a complexity of $O(m^2N^2)$. Note that this may vary depending on how the method is implemented. Our method involves sampling $m$ independent trajectories and ranking them. This process has a complexity of $O(mN^2)$. **Although ReAct is more efficient than our approach, our method performs much better.** **Our method is further more efficient and performs better than Reflexion based on the analysis above.**

Reviewer comment (Experiments):
What is the advantage of the LLM critic over a “learnt” critic which can take a policy rollout and provide a corresponding reward? Given that prospector is evaluated only in sim environment, why not use sim to learn such a critic?

Author response:
We would like to point out that this is one of the critics we consider in our paper. As we discuss in the last paragraph of section 3.2, **the in-context learning based critic fails to perform well in complex environments and a fine-tuned critic model performs better in this scenario.**

Reviewer comment (Novelty):
The authors dont seem to cite or mention self-refine: https://arxiv.org/pdf/2303.17651.pdf but that seemed very similar to self-asking in prospector too IMO.

Author response:
Thank you for pointing out this related work, we will include a discussion about the Self-Refine paper in our revision.

Reviewer comment (Reproducibility):
Opensourcing plans? Despite the simplicity of the approach, I encourage the authors to opensource their code for reproducibility.

Author response:
We plan to opensource our implementation upon paper acceptance.