Paper: Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds (ICLR-2024)

Reviewer comment (Evaluation):
The most important question for Steve-eye is the evaluation. Steve-eye is not evaluated on some long-horizon objectives like Voyager and GITM. With generated multi-modal instruction dataset, it is so intuitive that the model can perform well on the ENV-VC, FK-QA, SPP benchmarks as the instructions are generated for these tasks. The skill plan, visual caption or QA are actually studied on real-world images by some VLLM such as LLaVA. There is no reason to go back and evaluate VLLM on simulation environment just for such tasks. I think the main reason for the community to carry on research on game engine is to evaluate much more complex tasks that are not hard to attempt in real world. In this regard, I think the Steve-eye should have some evaluation manners as Voyager and GITM, to show what happens when we replace the environment information of game engine to visual perception.

Author response:
Thank you for your valuable suggestion, particularly regarding the supplementation of long-horizon tasks.

# 1. Long-horizon Evaluation Objectives

In fact, our work does involve long-horizon objectives in the skill planning benchmark (e.g., tasks of ``mining cobblestones to craft stone pickaxe'', ``crafting bed'' and ``harvest cooked beef'').
These tasks are typically broken down into dozens of skills and may require ten thousands of steps to complete.
More details of these long-horizon tasks are provided in **Table 6 in Appendix.2**.
In addition to these tasks, we conduct experiments on 10 new long-horizon objectives.
These objectives are iron-based task as detailed in **Table 7 in the Appendix.2**. 
As shown in **Table.R4** , our Steve-Eye consistently exhibits promising results across these long-horizon objectives, showing its robustness in handling complex game tasks.
In addition, we provide addition experiments in **Table R1, R2, R3** (in reponses to ijGQ) to demonstrate that visual perception is beneficial to text-only game engines.

**Table R4**: 10 long-horizon iron-based tasks.
| Model        | iron ingot | shears | iron bucket | iron-pickaxe | iron-axe | iron-sword | iron-shovel | tripwire hook | pressure plate | iron trapdoor |
|--------------|------------|--------|-------------|--------------|----------|------------|-------------|---------------|----------------|---------------|
| MineAgent    | 0.00       | 0.00   | 0.00        | 0.00         | 0.00     | 0.00       | 0.00        | 0.00          | 0.00           | 0.00          |
| gpt-assistant| 0.20       | 0.00   | 0.00        | 0.03         | 0.00     | 0.00       | 0.03        | 0.00          | 0.03           | 0.00          |
| ours         | 0.23       | 0.13   | 0.10        | 0.13         | 0.07     | 0.03       | 0.23        | 0.10          | 0.03           | 0.00          |


Thank you for your further reply!
In response to your concerns, we have conducted additional analysis and hope that the following detailed response will address your questions: 

# 1. Evaluation on metrics like Voyager and GITM
We understand your concern about our evaluation for skill planning. However, we respectfully argue that the evaluation of Steve-Eye for **your long-horizon criteria** is beyond the focus of our paper.

As we claim our contributions in the introduction section are threefold: instruction dataset, large multimodal model and training, and benchmarks including environmental visual captioning, foundational knowledge question answering, and skill prediction and planning. We can see that skill planning is a small fraction of our contributions. For Voyager and GITM, their work mainly focuses on skill planning, and they leverage the much more powerful GPT-4. Thus, they evaluate "the number of items unlocked, the extent of exploration range." However, this is not the case for Steve-Eye, since it is merely based on Llama-2-13B. Moreover, for skills, Voyager uses the generated code for Mineflayer API as skills, GITM uses manually structured skills, and Steye-Eye uses RL-learned policies as skills. Thus, it is unfair to evaluate Steye-Eye on their metrics, and we honestly admit that our current Steve-Eye will certainly underperform Voyager and GITM there. *In short, we believe our evaluation supports our claim on skill planning and it is unnecessary to evaluate Steve-Eye on your long-horizon criteria*.

# 2. Concern about evaluation which is still under the generated instruction-follow dataset.

Firstly, it's important to mention that we set our agent's birthplace in Minedojo using a seed value=500 during both data sampling and evaluation phases. 
The large seed value leads to spawning in a wide variety of locations within the map, including forests, plains, and lakes.
Due to the diverse range of birthplace, our instruction-following dataset is relatively limited in scale, meaning it is far enough from including all possible, unseen scenarios.

Secondly, for the FK-QA benchmark, the evaluation QA pairs are generated by ChatGPT based on Minecraft knowledge from Wiki. 
It's important to note that these pairs are distinct from our instruction dataset and are not utilized in the pre-training phase.

Lastly, to further demonstrate Steve-Eye can learn game-playing strategies beyond the confines of the instruction-follow dataset.
As shown in Table R5, we carry out additional experiments on 10 long-horizon iron-based tasks without their skill-related interaction instruction for training (ours zero-shot). 
Since the skill and plans of these unseen tasks are not included in the training dataset, we believe these experiments effectively demonstrate Steve-Eye's ability beyond the scope of the collected instructions.
Although performing worse, we observe that our Steve-Eye can still complete these challenging tasks and outperforms text-only methods driven by ChatGPT (gpt-assistant).

Table R5: 10 long-horizon iron-based tasks.
| Model           | iron ingot | shears | iron bucket | iron-pickaxe | iron-axe | iron-sword | iron-shovel | tripwire hook | pressure plate | iron trapdoor |
|--|---|---|----|----|-----|-----|---|--|-|----|
| MineAgent    | 0.00       | 0.00   | 0.00   | 0.00  | 0.00     | 0.00       | 0.00        | 0.00   | 0.00    | 0.00          |
| gpt-assistant   | 0.20       | 0.00   | 0.00    | 0.03    | 0.00     | 0.00       | 0.03   | 0.00  | 0.03   | 0.00          |
| ours zero-shot  | 0.17       | 0.10   | 0.13   | 0.07 | 0.03     | 0.03       | 0.17    | 0.03   | 0.00      | 0.00          |
| ours            | 0.23       | 0.13   | 0.10        | 0.13  | 0.07     | 0.03       | 0.23    | 0.10       | 0.03   | 0.00          |

Reviewer comment (Evaluation):
The paper presents Steve-Eye, an innovative integration of visual perception with Large Language Models (LLMs) for embodied agents. However, there’s a discernible lack of comprehensive discussion and analytical depth in the aspect of skill planning, particularly when compared to other game-playing agents such as “MineAgent.” While Steve-Eye showcases commendable performance, a more exhaustive exploration of its strategic capabilities, especially in comparison to MineAgent, is crucial. This detailed analysis could unveil the underlying reasons behind Steve-Eye’s superior performance, providing readers with substantial insights and a clearer understanding of its capabilities in the embodied task of planning.

Author response:
Thank you for your valuable suggestions! 
In this refined analysis, we take a closer look at skill planning and identify several key factors contributing to the standout performance of Steve-Eye compared to other game agents like MineAgent:

* **Adaptive Skill Planning Ability**:
MineAgent employs a multi-task policy learned through RL, which can be seen as relying on static, predefined plans to accomplish each task. 
In contrast, Steve-Eye adopts a different approach by decomposing a task into individual executable skills, forming new skill plans based on current, real-time conditions. 
This method endows Steve-Eye with exceptional adaptability in various gaming environments, enabling it to devise more effective and executable skill plans.
To validate this advantage, we conduct experiments (referred to as "ours w/o decomp") in **Tables R1, R2, and R3** below, which reveal a significant decline in performance without task decomposition.
The results indicates that integrating task decomposition is beneficial for solving complex tasks with basic skills trained via RL.

* **Additional Visual Understanding Ability**:
Our Steve-Eye directly leverages visual cues to assess its current status and respond accordingly.
As shown in **Table 5** in our main paper, visual cues empower Steve-Eye to outperforms text-only models like ``gpt-assistant'' even based on a much ligher-weight LLM.
Our work also suggests a universal application of visual cues in gaming agents, moving beyond the traditional reliance on rule-based indicators for status assessment.

* **More Effective In-context Reasoning Ability**: 
As depicted in **Table 3** in our main paper, Steve-Eye outperforms the vanilla Llama-2-7b by 6.4\% on the foundational knowledge QA benchmark.
Remarkably, it even exceeds GPT-Turbo-3.5, highlighting its acquisition of world knowledge.
Therefore, Steve-Eye has enhanced adaptability to environmental changes when planning skills.

**Table R1**: 7 tasks involving the process of ``cutting trees to craft primary items''
| Model      | stick | table | bowl | chest | trapdoor | sign | wooden pickaxe |
|------------|-------|-------|------|-------|----------|------|----------------|
| MineAgent  | 0.00  | 0.00  | 0.00 | 0.21  | 0.0      | 0.05 | 0.0            |
| Plan4MC    | 0.30  | 0.30  | 0.47 | 0.23  | 0.37     | 0.43 | 0.53           |
| ours w/o decomp | 0.03  | 0.03  | 0.23 | 0.13  | 0.03     | 0.07 | 0.00           |
| ours w/o ENV-VC | 0.27  | 0.27  | 0.33 | 0.43  | 0.37     | 0.27 | 0.40           |
| ours w/o FK-QA  | 0.37  | 0.23  | 0.40 | 0.47  | 0.27     | 0.33 | 0.37           |
| ours       | 0.40  | 0.30  | 0.43 | 0.53  | 0.33     | 0.37 | 0.43           |


**Table R2**: 7 tasks involving the process of ``mining cobblestones to craft advanced items''
| Model     | furnace | stonestairs | stoneslab | cobblestonewall | lever | torch | stonepickaxe |
|--|-|-|-|---|-------|-------|---------|
| MineAgent | 0.00    | 0.03   | 0.00      | 0.00      | 0.00  | 0.00  | 0.00    |
| Plan4MC   | 0.37    | 0.47   | 0.53      | 0.57      | 0.10  | 0.37  | 0.17    |
| ours w/o decomp| 0.13    | 0.23   | 0.17      | 0.13      | 0.03  | 0.07  | 0.00    |
| ours w/o ENV-VC| 0.23    | 0.30   | 0.33      | 0.37      | 0.27  | 0.23  | 0.23    |
| ours w/o FK-QA | 0.33    | 0.37   | 0.43      | 0.37      | 0.23  | 0.10  | 0.17    |
| ours      | 0.30    | 0.43   | 0.47      | 0.47      | 0.40  | 0.13  | 0.23    |

**Table R3**: 10 tasks involving the process of ``interacting with mobs to harvest food and materials''
| Model     | milkbucket | wool | beef | mutton | bed | painting | carpet | itemframe | cookedbeef | cookedmutton |
|-|-|-|-|-|-|-|-|-|-|-|
| MineAgent | 0.46   | 0.50 | 0.33 | 0.35   | 0.0 | 0.0   | 0.06   | 0.0       | 0.0  | 0.0    |
| Plan4MC   | 0.83   | 0.53 | 0.43 | 0.33   | 0.17| 0.13  | 0.37   | 0.07      | 0.20 | 0.13   |
| ours w/o decomp| 0.50   | 0.43 | 0.23 | 0.13   | 0.03| 0.03  | 0.17   | 0.03      | 0.00 | 0.03   |
| ours w/o ENV-VC| 0.67   | 0.43 | 0.33 | 0.27   | 0.13| 0.07  | 0.27   | 0.07      | 0.23 | 0.03   |
| ours w/o FK-QA | 0.63   | 0.53 | 0.43 | 0.37   | 0.17| 0.03  | 0.37   | 0.13      | 0.10 | 0.00   |
| ours      | 0.73   | 0.67 | 0.47 | 0.33   | 0.23| 0.07  | 0.43   | 0.10      | 0.17 | 0.07   |


The results in  **Table R1, R2, R3** also validate the association between our ENV-VC, FK-QA benchmarks and the improvement of decision-making process.
When the ENV-VC task is removed, and we directly utilize visual cues without further instruction tuning, there's a notable drop in performance.
This clearly demonstrates the critical role of ENV-VC task in enhancing the model to intepret and utilize visual cues effectively.
Similarly, the exclusion of the FK-QA task yields a performance decrease, emphasizing the essential contribution of these two pre-training tasks in enhancing our model's decision-making capabilities.

Reviewer comment (Evaluation):
The paper discusses two other tasks: Environmental Visual Captioning (ENV-VC) and Foundational Knowledge Question Answering (FK-QA). While these tasks are undoubtedly interesting and add value to the paper, their direct connection to the agent’s in-game execution and decision-making processes is not explicitly clear. Strengthening this connection and elaborating on how these tasks intricately weave into the agent’s planning and action sequences would significantly enhance the paper’s overall contribution to the field.

Author response:
Thank you for your valuable suggestions! 
In this refined analysis, we take a closer look at skill planning and identify several key factors contributing to the standout performance of Steve-Eye compared to other game agents like MineAgent:

* **Adaptive Skill Planning Ability**:
MineAgent employs a multi-task policy learned through RL, which can be seen as relying on static, predefined plans to accomplish each task. 
In contrast, Steve-Eye adopts a different approach by decomposing a task into individual executable skills, forming new skill plans based on current, real-time conditions. 
This method endows Steve-Eye with exceptional adaptability in various gaming environments, enabling it to devise more effective and executable skill plans.
To validate this advantage, we conduct experiments (referred to as "ours w/o decomp") in **Tables R1, R2, and R3** below, which reveal a significant decline in performance without task decomposition.
The results indicates that integrating task decomposition is beneficial for solving complex tasks with basic skills trained via RL.

* **Additional Visual Understanding Ability**:
Our Steve-Eye directly leverages visual cues to assess its current status and respond accordingly.
As shown in **Table 5** in our main paper, visual cues empower Steve-Eye to outperforms text-only models like ``gpt-assistant'' even based on a much ligher-weight LLM.
Our work also suggests a universal application of visual cues in gaming agents, moving beyond the traditional reliance on rule-based indicators for status assessment.

* **More Effective In-context Reasoning Ability**: 
As depicted in **Table 3** in our main paper, Steve-Eye outperforms the vanilla Llama-2-7b by 6.4\% on the foundational knowledge QA benchmark.
Remarkably, it even exceeds GPT-Turbo-3.5, highlighting its acquisition of world knowledge.
Therefore, Steve-Eye has enhanced adaptability to environmental changes when planning skills.

**Table R1**: 7 tasks involving the process of ``cutting trees to craft primary items''
| Model      | stick | table | bowl | chest | trapdoor | sign | wooden pickaxe |
|------------|-------|-------|------|-------|----------|------|----------------|
| MineAgent  | 0.00  | 0.00  | 0.00 | 0.21  | 0.0      | 0.05 | 0.0            |
| Plan4MC    | 0.30  | 0.30  | 0.47 | 0.23  | 0.37     | 0.43 | 0.53           |
| ours w/o decomp | 0.03  | 0.03  | 0.23 | 0.13  | 0.03     | 0.07 | 0.00           |
| ours w/o ENV-VC | 0.27  | 0.27  | 0.33 | 0.43  | 0.37     | 0.27 | 0.40           |
| ours w/o FK-QA  | 0.37  | 0.23  | 0.40 | 0.47  | 0.27     | 0.33 | 0.37           |
| ours       | 0.40  | 0.30  | 0.43 | 0.53  | 0.33     | 0.37 | 0.43           |


**Table R2**: 7 tasks involving the process of ``mining cobblestones to craft advanced items''
| Model     | furnace | stonestairs | stoneslab | cobblestonewall | lever | torch | stonepickaxe |
|--|-|-|-|---|-------|-------|---------|
| MineAgent | 0.00    | 0.03   | 0.00      | 0.00      | 0.00  | 0.00  | 0.00    |
| Plan4MC   | 0.37    | 0.47   | 0.53      | 0.57      | 0.10  | 0.37  | 0.17    |
| ours w/o decomp| 0.13    | 0.23   | 0.17      | 0.13      | 0.03  | 0.07  | 0.00    |
| ours w/o ENV-VC| 0.23    | 0.30   | 0.33      | 0.37      | 0.27  | 0.23  | 0.23    |
| ours w/o FK-QA | 0.33    | 0.37   | 0.43      | 0.37      | 0.23  | 0.10  | 0.17    |
| ours      | 0.30    | 0.43   | 0.47      | 0.47      | 0.40  | 0.13  | 0.23    |

**Table R3**: 10 tasks involving the process of ``interacting with mobs to harvest food and materials''
| Model     | milkbucket | wool | beef | mutton | bed | painting | carpet | itemframe | cookedbeef | cookedmutton |
|-|-|-|-|-|-|-|-|-|-|-|
| MineAgent | 0.46   | 0.50 | 0.33 | 0.35   | 0.0 | 0.0   | 0.06   | 0.0       | 0.0  | 0.0    |
| Plan4MC   | 0.83   | 0.53 | 0.43 | 0.33   | 0.17| 0.13  | 0.37   | 0.07      | 0.20 | 0.13   |
| ours w/o decomp| 0.50   | 0.43 | 0.23 | 0.13   | 0.03| 0.03  | 0.17   | 0.03      | 0.00 | 0.03   |
| ours w/o ENV-VC| 0.67   | 0.43 | 0.33 | 0.27   | 0.13| 0.07  | 0.27   | 0.07      | 0.23 | 0.03   |
| ours w/o FK-QA | 0.63   | 0.53 | 0.43 | 0.37   | 0.17| 0.03  | 0.37   | 0.13      | 0.10 | 0.00   |
| ours      | 0.73   | 0.67 | 0.47 | 0.33   | 0.23| 0.07  | 0.43   | 0.10      | 0.17 | 0.07   |


The results in  **Table R1, R2, R3** also validate the association between our ENV-VC, FK-QA benchmarks and the improvement of decision-making process.
When the ENV-VC task is removed, and we directly utilize visual cues without further instruction tuning, there's a notable drop in performance.
This clearly demonstrates the critical role of ENV-VC task in enhancing the model to intepret and utilize visual cues effectively.
Similarly, the exclusion of the FK-QA task yields a performance decrease, emphasizing the essential contribution of these two pre-training tasks in enhancing our model's decision-making capabilities.

Reviewer comment (Experiments):
The inclusion of “gpt-assistant” as a baseline in the performance evaluation, particularly noted in Table 5, brings about certain ambiguities. The choice of baselines is critical, and in this context, one might wonder if “Voyager” or other models specifically tailored for the Minecraft environment would serve as more apt comparisons. By opting for baselines that are more closely aligned with the Minecraft gaming milieu, the paper could present a more robust and convincing argument for Steve-Eye’s effectiveness.

Author response:
For the "gpt-assistant" baseline, we develop an interactive planning framework using ChatGPT. 
In this setup, ChatGPT serves as the planner, formulating skill plans in response to prompts that include task descriptions and environmental observations. 
This approach involves chain-of-thoughts prompting, where we initially provide the planner with a few-shot demonstration with explanations to guide the initial planning step.
To address frequent errors observed during testing, we incorporate specific planning rules into the prompts. 
As the planning process progresses, the planner encounters various scenarios: invalid skill names, redundant skills (already executed), successful skill execution, or failed skill execution. 
For each of these cases, we carefully craft language feedback and ask the planner to re-plan based on inventory changes.

In addition to ``gpt-assistant'', we conduct experiments to compare with another game planning approach for Minecraft: Plan4MC [1].
Similar to our Steve-Eye, this approach utilizes a similar strategy to break down a given Minecraft task into several skills, and execute these skills one by one.
By employing a specialized skill graph, Plan4MC guarantees the generation of executable plans with 100\% certainty, thereby serving as an upper bound for our approach.
As shown in **Table R1, R2, R3** earlier, our Steve-Eye presents competitive performance when compared to Plan4MC, 
This is noteworthy considering Steve-Eye uses a significantly less optimal LLM (LLaMA-2) compared to ChatGPT used by Plan4MC.

In our analysis, we chose not to compare our Steve-Eye with methods like Voyager, as there are significant differences in their underlying mechanisms.
Voyager controls the agent by generating code for the Mineflayer API using GPT-4, a process that is inherently different from Steve-Eye, which is driven by a considerably lighter-weight LLM and employs pre-defined RL policies for each skill control. 
Additionally, Voyager heavily relies on curated prompting for its planning and skill selection process, a feature not as prominently utilized in our approach.
Given these fundamental differences, a direct comparison between Steve-Eye and Voyager would not be equitable. 
However, it's crucial to recognize that Steve-Eye is compatible with Voyager and other similar methods. 
Voyager uses LLMs to translate an agent's observations into text, while Steve-Eye excels at interpreting multimodal cues. 
This capability positions Steve-Eye as a potential multimodal foundation model to replace the LLMs in methods like Voyager.


[1] Plan4MC: Skill Reinforcement Learning and Planning for Open-World Minecraft Tasks

Reviewer comment (Writing):
As depicted in Figure 4 and discussed in various sections of the paper, Steve-Eye has the capability to generate image outputs. However, the practical application and utility of these image outputs in the context of the tasks at hand appear somewhat nebulous. Providing readers with a clearer exposition on how these image outputs can be leveraged for specific tasks would enhance the paper’s clarity and comprehensibility.

Author response:
Thank you for this suggestion.

Firstly, integrating visual generation into our pre-training introduces a novel task that significantly broadens the scale and diversity of our instruction-following dataset.
This expansion helps prevent oversimplification of the training process and ensures that the model consistently produces accurate and coherent responses, especially when dealing with complex environments.

Secondly, as discussed in our introduction, we argue that multimodal outputs offer a more intuitive way to represent the world, leading to an effective interface for interaction with human users. 
Therefore, although the multimodal output in this work is not generated for specific tasks, 
we believe that this is crucial for developing a general-purpose game assistant.

Reviewer comment (Presentation):
My initial interaction with Figure 1 was marked by confusion, necessitating several revisits to fully grasp its intended message. A visual element such as a horizontal separator line could significantly improve the figure’s readability, distinctly delineating the unfavorable responses from the preferable ones. Currently, the figure could be misinterpreted as a continuous chat, leading to potential misunderstanding. Implementing this small yet impactful change would streamline the reader’s experience, fostering a quicker and clearer comprehension of the depicted scenarios.

Author response:
Thank you for your suggestion. 
We have improved the readability of our figures and address the minor issues in our revised version.