Paper: Adaptive Environmental Modeling for Task-Oriented Language Agents (ICLR-2024)

Reviewer comment (Evaluation):
The performance metric "score" is not clearly defined and the method is not clear and not objective.

Author response:
For specific tasks like Webshop and ALFWorld-Eco, the scores are typically based on conventional accuracy or , in other words, success rates. Your query seems to center around the Score for Comprehensiveness. Let’s explore this in detail: Given your expertise in reinforcement learning, you can think of this Score as akin to a weighted State Coverage. The 'weight' in this context is indicative of the informational value of each action in enriching our understanding of the environment from a human perspective. Take the Webshop example: for the observation [result page 1], which includes [result 1]  (a), [result 2] (b), [result 3] (c), [Back to Search] (d), and [Next >] (e). The letters in "( )" representing the potential informational value of each clickable action. Assuming resources allow the agent to explore two actions, an agent 1 opts to explore [result 1] and [Next >], its score would be calculated as $\frac{a + e}{a + b + c + d + e} $. Conversely, if another agent 2 explores [result 1] and [result 2], its score would be $\frac{a + b}{a + b + c + d + e} $. 

In this context, for question E, absent a human-centric evaluation and assuming equal weights for all actions, both agents 1 and 2 would end up with the same score of $\frac{2}{5} $, thereby diminishing the significance of the comprehensiveness evaluation. To address the issue of potential human bias, we have meticulously developed a set of Scoring Criteria for Comprehensiveness. 
- **Information Relevance**: Evaluate how relevant the information at each node is to the overall understanding of the Webshop environment. Irrelevant or overly detailed information about specific products should be scored lower.
- **Depth of Environmental Insight**: Score based on how deeply the node's information contributes to understanding the environment. Nodes providing unique or essential insights should receive higher scores.
- **Clarity and Conciseness**: Assess whether the information is clear and to the point. Unnecessarily verbose or unclear descriptions should be scored lower.

We also incorporate assessments from multiple crowd-sourced evaluators to ensure a balanced perspective, while paying close attention to the consistency of the scoring system.

**Regarding consistency**: To assess score distribution consistency, we first conducted an intuitive check and found that the score distribution was relatively consistent. We then proceeded with a quantitative statistical analysis. To minimize the influence of individual crowd workers' scoring preferences, we normalized the scores for each worker. This involved dividing each node's score by the total node scores and then multiplying by 100. Subsequently, for each node, we calculated the standard deviation of scores given by different crowd workers. Finally, we computed the average of these standard deviations across all nodes. The results were 0.76 for Webshop and 0.42 for ALFWorld-Eco, indicating a relative consistency in the crowd workers' scoring.

Reviewer comment (Experiments):
The self-reference method described in "we propose employing the agent itself to evaluate states" is fundamentally flawed since the underlying LLM is not perfect, and there is no study about how different this may deviate from a perfect model.

Author response:
**1.The self-reference method seems fundamentally flawed, given that the underlying Large Language Model (LLM) is not perfect.**

It's important to highlight that Algorithm 1 is responsible for building the Action-Observation Tree, with the LLM being just a component of this process. We have taken into account your concern about the LLM's imperfections. The design of Algorithm 1 greatly lessens the dependency on the LLM's performance, thereby reducing the risk of potential flaws, such as hallucinations, influencing the outcomes. The role of the language model is simply to choose the most exploratory action from a given set of observations and executable actions. Thus, the language model is more than capable of performing its designated role within the algorithm. This is evidenced in our experiments, as described in Section 4.3.1, where it is noted that “agents developed with both GPT-3.5 and GPT-4 frameworks succeed in generating nearly 100% executable actions within both environments”. 

**2.There's a lack of study on how this might differ from a perfect model.**

As detailed in Section 4.3.1 of our experiments, the language model consistently produces 100% executable actions when presented with observations and a set of executable actions. This indicates that the Action-Observation Trees constructed by Algorithm 1 are inherently accurate. The primary aspect of their imperfection is not incorrectness, but incompleteness. The Comprehensiveness score in Section 4.3.1 assesses how far these trees are from being complete Action-Observation Trees.

Reviewer comment (Evaluation):
How to score is not clearly defined.

Author response:
For specific tasks like Webshop and ALFWorld-Eco, the scores are typically based on conventional accuracy or , in other words, success rates. Your query seems to center around the Score for Comprehensiveness. Let’s explore this in detail: Given your expertise in reinforcement learning, you can think of this Score as akin to a weighted State Coverage. The 'weight' in this context is indicative of the informational value of each action in enriching our understanding of the environment from a human perspective. Take the Webshop example: for the observation [result page 1], which includes [result 1]  (a), [result 2] (b), [result 3] (c), [Back to Search] (d), and [Next >] (e). The letters in "( )" representing the potential informational value of each clickable action. Assuming resources allow the agent to explore two actions, an agent 1 opts to explore [result 1] and [Next >], its score would be calculated as $\frac{a + e}{a + b + c + d + e} $. Conversely, if another agent 2 explores [result 1] and [result 2], its score would be $\frac{a + b}{a + b + c + d + e} $. 

In this context, for question E, absent a human-centric evaluation and assuming equal weights for all actions, both agents 1 and 2 would end up with the same score of $\frac{2}{5} $, thereby diminishing the significance of the comprehensiveness evaluation. To address the issue of potential human bias, we have meticulously developed a set of Scoring Criteria for Comprehensiveness. 
- **Information Relevance**: Evaluate how relevant the information at each node is to the overall understanding of the Webshop environment. Irrelevant or overly detailed information about specific products should be scored lower.
- **Depth of Environmental Insight**: Score based on how deeply the node's information contributes to understanding the environment. Nodes providing unique or essential insights should receive higher scores.
- **Clarity and Conciseness**: Assess whether the information is clear and to the point. Unnecessarily verbose or unclear descriptions should be scored lower.

We also incorporate assessments from multiple crowd-sourced evaluators to ensure a balanced perspective, while paying close attention to the consistency of the scoring system.

**Regarding consistency**: To assess score distribution consistency, we first conducted an intuitive check and found that the score distribution was relatively consistent. We then proceeded with a quantitative statistical analysis. To minimize the influence of individual crowd workers' scoring preferences, we normalized the scores for each worker. This involved dividing each node's score by the total node scores and then multiplying by 100. Subsequently, for each node, we calculated the standard deviation of scores given by different crowd workers. Finally, we computed the average of these standard deviations across all nodes. The results were 0.76 for Webshop and 0.42 for ALFWorld-Eco, indicating a relative consistency in the crowd workers' scoring.

Reviewer comment (Evaluation):
The method described in "we adopted a manual evaluation method, specifically utilizing a crowdsourcing approach, allowing ten individuals familiar with the relevant environment to score the amount of information contained in the environmental observations corresponding to each action, with a total score of 100" appears very vague, subjective and arbitrary.

Author response:
For specific tasks like Webshop and ALFWorld-Eco, the scores are typically based on conventional accuracy or , in other words, success rates. Your query seems to center around the Score for Comprehensiveness. Let’s explore this in detail: Given your expertise in reinforcement learning, you can think of this Score as akin to a weighted State Coverage. The 'weight' in this context is indicative of the informational value of each action in enriching our understanding of the environment from a human perspective. Take the Webshop example: for the observation [result page 1], which includes [result 1]  (a), [result 2] (b), [result 3] (c), [Back to Search] (d), and [Next >] (e). The letters in "( )" representing the potential informational value of each clickable action. Assuming resources allow the agent to explore two actions, an agent 1 opts to explore [result 1] and [Next >], its score would be calculated as $\frac{a + e}{a + b + c + d + e} $. Conversely, if another agent 2 explores [result 1] and [result 2], its score would be $\frac{a + b}{a + b + c + d + e} $. 

In this context, for question E, absent a human-centric evaluation and assuming equal weights for all actions, both agents 1 and 2 would end up with the same score of $\frac{2}{5} $, thereby diminishing the significance of the comprehensiveness evaluation. To address the issue of potential human bias, we have meticulously developed a set of Scoring Criteria for Comprehensiveness. 
- **Information Relevance**: Evaluate how relevant the information at each node is to the overall understanding of the Webshop environment. Irrelevant or overly detailed information about specific products should be scored lower.
- **Depth of Environmental Insight**: Score based on how deeply the node's information contributes to understanding the environment. Nodes providing unique or essential insights should receive higher scores.
- **Clarity and Conciseness**: Assess whether the information is clear and to the point. Unnecessarily verbose or unclear descriptions should be scored lower.

We also incorporate assessments from multiple crowd-sourced evaluators to ensure a balanced perspective, while paying close attention to the consistency of the scoring system.

**Regarding consistency**: To assess score distribution consistency, we first conducted an intuitive check and found that the score distribution was relatively consistent. We then proceeded with a quantitative statistical analysis. To minimize the influence of individual crowd workers' scoring preferences, we normalized the scores for each worker. This involved dividing each node's score by the total node scores and then multiplying by 100. Subsequently, for each node, we calculated the standard deviation of scores given by different crowd workers. Finally, we computed the average of these standard deviations across all nodes. The results were 0.76 for Webshop and 0.42 for ALFWorld-Eco, indicating a relative consistency in the crowd workers' scoring.

Reviewer comment (Reproducibility):
In the second stage of environment update, no details were provided on how to interactively generate new trajectories and how to update the environment tree using the new trajectories, such as the corresponding prompt.

Author response:
Thank you for taking the time to review this paper and provide detailed feedback. We sincerely appreciate the insightful comments and the critical examination of our work. Below we provide responses to the concerns and questions you have raised:

### **Reply to Weaknesses1: Provide details about the usage of the language model and the prompt.**

The language model is primarily used as an Evaluator in the initial exploratory phase  and Task Executor in the second stage. 

**1.Evaluator**: As illustrated in Algorithm 1, in the step of actions_next ← A.SelectActions(node.observation, actions_available), given the observation and the available actions set, the action that maximizes the exploration of the environment is selected by the language model. The specific prompt is essentially as follows:
```
Activity: You are an agent engaged in exploring the Webshop site.

Objective: Maximize environmental exploration by selecting appropriate actions, through iteratively performing the Think and Action steps outlined below:

Think: Given the observation [ the observation details ], consider which actions would maximize the acquisition of environmental information.

Action: Based on your formulated strategies, choose the top-k appropriate actions to interact with the environment.

The actions you take must be one of the following:
"think[strategy]" to accomplish the Objective.
"search[product]" to look for specific products.
"click[button]" to interact with actionable elements. Only strings in "[ ]" are buttons.
```
**2.Task Executor**: 
```
Activity: You are an agent and engaged in an online shopping task on the Webshop site. 

Environment：Below are the interaction for the Webshop : [Action-Observation Tree](Similar to figure 3,4,5 in appendix)

Objective: Complete the purchasing task concerning 'the instruction' by iteratively performing the Think and Action steps outlined below:

Think: Given the observation [ the observation details ], think how you can accomplish the instruction task.

Action: Based on your formulated strategies, perform an action to interact with the environment.
 
The actions you take must be one of the following:
"think[strategy]" to accomplish the instruction task.
"search[product]" to look for specific products.
"click[button]" to interact with actionable elements. Only strings in "[ ]" are button.
```
Then you can set the instruction task.

**3. Updating the Environment Tree with New Trajectories.**

As the Agent carries out its tasks, it concurrently constructs an Action-Observation Tree, essentially forming new trajectories. Thus, the update as defined in Formula 3 fundamentally represents a fusion of two distinct trees. Notably, this phase proceeds without the  language model. We meticulously engage in a breadth-first traversal of the nodes within the trajectory-representing tree. Whenever we encounter nodes absent in the tree that depicts the environment, we refine and expand the latter by integrating these new nodes. To illustrate, refer to the comparative analysis of the Action-Observation Trees in figures 4 and 5 in the appendix. Figure 5 vividly demonstrates the enrichment of the tree with novel nodes, such as [Buy Now] and [Attributes], introduced through the progression of task trajectories.