Paper: GROOT: Learning to Follow Instructions by Watching Gameplay Videos (ICLR-2024)

Reviewer comment (Evaluation):
Figures 5 and 6 have rather high error bars which could indicate that results are not significantly different (see questions). Also, there are some questions around the evaluation and inference setup (see questions), which reduces my confidence in the results. However, overall, I believe the results hold and find the proposed architecture beneficial.

Author response:
Thank you for raising this concern. We have updated the number of experiments to $N=10$ in the caption.

The larger variance is related to the environmental characteristics of Minecraft. Taking the task of "collect wood" as an example, we summoned the agent in a forest with complex terrain, including pools, caves, etc. We observed that when the agent accidentally fell into a pool, it took a lot of time to climb out of the water. Since the video instruction for chopping wood did not include a description of how to guide the agent to climb out of the pool, in most cases, the agent remained stuck in the pool and was thus unable to obtain any wood. If the agent did not fall into the pool, it could usually chop down a tree and obtain 4-5 blocks of wood at once, resulting in a larger variance. This phenomenon is also reflected in the STEVE-1 paper, such as its Figures 3, 4, 5, 7.

Reviewer comment (Experiments):
Can the authors validate the controller's generalization ability for diverse tasks by providing relevant video instructions for unseen tasks and skills? This would be of great significance for the integration of LLM-based planning agent capabilities.

Author response:
### **(Weakness 1, Part 1) Generalization ability.** 
> Can the authors validate the controller's generalization ability for diverse tasks by providing relevant video instructions for unseen tasks and skills? This would be of great significance for the integration of LLM-based planning agent capabilities.

Thank you for your question. Our training method is somewhat similar to GPT. OpenAI did not need to define explicit tasks or categorize their collected data by task when pre-training GPT. Instead, they crawled a large amount of noisy text data from the internet and trained the model's ability to predict the next word autoregressively. After training, GPT has the ability to solve "unseen" tasks in zero-shot. Similarly, in the training process of GROOT, we do not need to define explicit tasks or cut each video into clips according to their semantics. Instead, we train the model's ability to self-imitate on a video segment. After training, by providing GROOT with a gameplay video that describes how to complete a task, it can imitate the process described in the video to complete the specified task in Minecraft. 

We can understand the tasks defined in Minecraft SkillForge as those that occur after GROOT has completed training, meaning that GROOT actually knows how to perform far more tasks than those included in SkillForge. As videos can be obtained from the internet in an unsupervised manner, we may not be able to find a behavior that GROOT has never seen, just as we would have difficulty finding a task that GPT has never encountered. 

Here, we highlight two tasks that are rarely seen in the dataset: "build snow golems" and "dig down three and fill one up". The former is only known to a few players, while the latter is only used occasionally by very beginner players. GROOT achieved a success rate of 60% in the "build snow golems" task and 30% in the "dig three down and fill one up" task, which demonstrates the extent GROOT is able to perform on long-tailed or even "OOD" tasks.

Reviewer comment (Reproducibility):
If GROOT is combined with a high-level planner, how would it construct video instructions?

Author response:
### **(Weakness 1, Part 2) Ideas of constructing video instructions when combined with a high-level planner.** 
Thank you for raising this question. In recent years, there have been a number of works [1, 2] aimed at high-level planning in Minecraft. These works unanimously suggest that a low-level controller should construct a skill library for use by higher-level planners. For example, DEPS [1] uses a pre-trained goal-conditioned policy as the underlying controller, as provided in [5]. However, training such a goal-conditioned policy still requires a large amount of high-quality, task-specific demonstrations. For GROOT, we can use relevant keywords to search on YouTube or ask Minecraft players to provide some tutorials for the skills that may be called by higher-level models. In practice, we can collect about 5 gameplay videos for each skill then select the best performing one that GROOT can imitate, save it in the database. It is easy to retrieve the gameplay video from the database when the planner needs to call some skills. This has greatly reduced the cost of building a skill library. 
**We have updated the relevant discussion in Appendix J for interested readers to check.**

[1] Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents
[2] Ghost in the Minecraft: Generally Capable Agents for Open-World Enviroments via Large Language Models with Text-based Knowledge and Memory
[3] Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction

Reviewer comment (Reproducibility):
For the results in Figure 7, how are the two video instructions of GROOT specifically input to the controller? Is it directly combined and input, or is the video switched manually after reaching the specified depth?

Author response:
### **(Weakness 1, Part 3) Clarification of the results in Figure7.** 
Thank you for your question. The two video instructions are completely independent of each other, and there is no concatenating operation between them. (If the experiments in Section 5.2 have caused any confusion, we apologize. The experiments in Section 5.2 are irrelevant to the experiments in Section 5.3.) We used a very simple logical discrimination program. At the beginning, we passed the video segment "dig down" as an instruction to GROOT(or STEVE-1), which would perform the behavior of digging down. The program would dynamically judge the agent's current height. (Here, obtaining height information is reasonable even human players needs to use the F3 function to check their current height when searching for diamonds.) When the player's height reached the 12 level, GROOT(or STEVE-1)'s input instruction was modified to "mine horizontally" (actually a gameplay video), which continued to guide the agent to explore and dig horizontally. Although the procedure that a high-level planner needs to execute in practical use may be quite complicated, it is not the emphasis of this paper. However, the way it interacts with the controller is roughly the same. We use such a simple logical discrimination program to demonstrate in the most simplified way how our GROOT works with the planner to solve long-sequence tasks.

Reviewer comment (Novelty):
Although the paper often emphasizes learning from gameplay videos, the inverse dynamic model (IDM) is based on VPT, and there is a lack of discussion on IDM. Therefore, I believe that the essence of this paper is to use task-related state sequences as instructions and train a goal-conditioned policy. Using state sequences as instructions/goals for training agents is not uncommon. Methods such as GATO[1], AD[2] directly input demonstrations as token sequences into transformers, while [3] uses VLM to extract video information. Even without imposing constraints on the goal space, agents can exhibit good performance. The authors should compare or discuss these methods.

Author response:
### **(Weakness 2, Part 1) Emphasize the importance of inverse dynamic model.** 
Thanks for raising this question. The Inverse Dynamic Model is indeed critical in our paper because it provides necessary behavior labels for video data to compute the behavior cloning loss in subsequent steps. **To avoid misunderstandings, we added a section to introduce inverse dynamic model in the Appendix C of the paper.** In addition, we have also emphasized this in the appropriate section of the paper, such as conclusion.

Fortunately, VPT [1] made an important and surprising discovery that we may only need to collect a small amount of supervised data with behavior labels to train a highly accurate inverse dynamic model, even in complex environments like Minecraft. In parallel, a work presented at ECCV2022 [2] trained an accurate IDM model in the field of autonomous driving, further validating the feasibility of training IDM models in complex domains. This allows us to migrate this algorithm to other complex open environments at a lower cost. 

 [1] Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., ... & Clune, J. (2022). Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35, 24639-24654.
 [2] Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining.

### **(Weakness 2, Part 2) Discussions of related works.**
Thanks for pointing out these related works, AD[1], Gato[2], and UniAgent[3].
We agree that the Transformer architecture has gradually become a common modeling method in decision control tasks. However, we believe that our method is conceptually different from GATO, AD, and UniAgent in terms of what we aim to learn/model.

As for the AD algorithm, it mainly focuses on how to use the Transformer to model the training process of reinforcement learning algorithms. It can be viewed as a meta-reinforcement learning algorithm. It requires a large number of historical trajectories generated during reinforcement learning training as well as reward signals related to the tasks. However, designing reward functions for each task in Minecraft is impractical because the number of tasks is endless. In addition, during the inference process, the Transformer needs to repeat execution with the environment many times to obtain a good policy, which is infeasible in some dangerous environments. For example, in Minecraft, a single death may lead to the entire game's progress being wiped out. Therefore, the AD algorithm is not suitable for solving the problem encountered in this paper.

As for the Gato model, although it has achieved good results in the multi-domain, multi-modal, and multi-task fields, its training still requires pre-defined tasks, which are learned by collecting a large amount of high-quality expert teaching data under the specific tasks. Actually, we do not have task labels for our gameplay videos. Therefore, GATO is not suitable for this environment.

Regarding the UniAgent model, it needs to interact with the environment in real-time, which restricts its ability to scale up to learning large-scale skills. We believe that this model can achieve good performance in life-long learning settings; however, the focus of this paper is to learn the open-ended instruction-following ability from a large number of offline video data. Therefore, UniAgent is not suitable for solving this problem. 

**We have taken your comment into account and put the related discussion in the "Related Works" section.**

[1] In-context reinforcement learning with algorithm distillation. 
[2] A generalist agent. 
[3] Towards a unified agent with foundation models.

Reviewer comment (Evaluation):
As shown in Figure 5, if the video contains multiple goals, the performance may be affected. Would this lead to more stringent requirements for video quality during application?

Author response:
### **(Weakness 3) Clarification of Figure 5.** 
Thanks for your question. The experiments in Figure 5 demonstrate a bonus discovery of our learned goal space: concatenating multiple goals enables GROOT to solve multiple tasks simultaneously to some extent. However, we also found that this may have the side effect of decreasing the performance of completing individual tasks. This is acceptable because GROOT was not trained on concatenated videos. Providing higher-quality data should lead to improved performance in the end. The discussion of how to further improve GROOT's performance on concatenated videos is beyond the scope of this manuscript and will be left to future work.

Reviewer comment (Evaluation):
In Figure 6, when the KL loss is ablated, the performance decreases. Can the authors provide a visualization of the goal space without KL loss to explain its contribution to the agent's generalization in the goal space? As a significant part of the main contribution, the KL loss needs to be analyzed in more detail.

Author response:
### **(Weakness 4) Visualization of learned goal space.**
Thank you for raising the question. We have updated the visualization of the goal space for GROOT with and without KL regularization in Figure 4. It can be seen that both of them demonstrate certain semantic understanding capabilities in clustering. The clustering effect is slightly better with KL regularization, but the difference is not very significant. Intuitively, KL loss is mainly used to bring the distribution of the posterior latent closer to the distribution of the prior latent, thereby enhancing the abstraction capability of the encoder and improving the generalization ability of the model. However, the generalization ability does not have a strong correlation with the clustering effect on the goal space of videos within the dataset, which is confirmed by our quantitative results, please refer to Figure 6.

Reviewer comment (Writing):
In Section 3, symbols like $\theta$ lack explicit definitions.

Author response:
- **Clarification of parameters $\theta, \phi$ in Section 3:** $\theta$ denotes the parameters for policy $\pi_\theta(a_t | s_{0:t}, g)$, prior $p_\theta(g | s_{0:t})$, and inverse dynamic model $p_\theta(a_{\tau} | s_{0:\tau+1})$. $\phi$ denotes the parameters for the posterior $q_\phi(g | s_{0:T})$.

Reviewer comment (Theory):
In Equation 13, why does log $p(a_τ|s_{0:τ})$ depend solely on the environment dynamics?

Author response:
- **Clarification of $p(a_\tau | s_{0:\tau})$:** Thank you very much for pointing this out. This was a typo on our part. It actually describes a distribution within the dataset that is irrelevant to the goal space and goal-conditioned policy that we wish to learn.

Reviewer comment (Experiments):
How is the segment length T set? Is there only a single goal within each segment?

Author response:
- **Clarification of segment length "T":** T is a hyperparameter that is related to the maximum context length that TransformerXL processes at one time. In this work, we set T to 128, to be consistent with the structure setting of VPT. On the one hand, this is due to limited computational resources, as a larger T would consume a considerable amount of GPU memory. On the other hand, a context length of 128 already has strong expressive power and can represent most short-horizon tasks. 

- **Is there only a single goal within each segment?** Each video segment of length T is considered a complete behavior, even if it contains multiple steps, and is encoded as a single goal embedding.

Reviewer comment (Experiments):
While the multi-step video representation for goals is more expressive than other prior Minecraft agents (that use language or outcome videos), having to provide a video to condition on can be difficult to do in practice if we do not already have access to similarly representative videos, and as the authors note, training the video-based goal space is challenging. On the other hand, it is much easier to describe a desired goal in language. It would be interesting to see if this method can be adapted flexibility to condition on either visual or text based goals, as in STEVE-1.

Author response:
Thank you for your question. We believe the video instruction space is more general and some behaviors are more naturally expressed with video. It is the focus of our paper. 

Meanwhile, we also agree on the importance of conditioning policy on text instructions. Although video instruction has strong expressiveness, it still requires preparing at least one gameplay video for a new task. For some tasks, such as collecting wood or stones, using natural language to specify a goal is a more natural approach. **We have conducted the text-conditioning experiment, please see Appendix I**. 

Considering that it is not feasible for us to collect text-video data with as much diversity as STEVE-1 within a short period of time, and we did not directly utilize the initial alignment space provided by MineCLIP during GROOT training. Therefore, we chose a new alignment method, which is to directly replace the video encoder with a BERT architecture text encoder (randomly initialized) and freeze the decoder. We optimize the text encoder using behavior cloning. 

We utilize the meta information in the contractor data to generate text-demonstration pairs. For example, in the task of "collect wood", we identify the moment $t$ when event "mine\_block:oak\_log" is triggered in the video, and we capture the frames within the range of $[t-127, t]$ to form a video clip, with "mine block oak log" assigned as its text, thus constructing a sample. Having been fine-tuned on these data, our model demonstrated some steerabilities in the text instruction space, as shown in the table below. (The term "baseline" refers to the model before being fine-tuned, while "fine-tuned" refers to the final model after fine-tuning. ) 

| Variant    | mine grass    | mine wood    | mine stone    | mine seagrass| pickup beef        | mine dirt  |
| ---------- | ------------- | ------------ | ------------- | ------------ | ---------------- | ---------- |
| baseline   | 3.9           | 0.4          | 1.8           | 1.3          | 0.0              | 0.0        |
| fine-tuned | 17.3          | 3.7          | 11.5          | 1.2          | 0.1              | 1.3        |

We find that the agent fine-tuned on the text-demonstration dataset shows a basic understanding of text instructions. Our method exhibits progress in tasks such as "mine grass", "mine wood", "mine stone", "mine seagrass", "pickup beef" and "mine dirt". However, it falls short in successfully completing tasks such as "mine seagrass". We speculate that this may be related to the distribution of the data, as there is much less data available for "mine seagrass" compared to the other tasks (only about 300 trajectories). 

We emphasize that this experiment is very preliminary. In this experiment, the steerability of the agent fine-tuned on text instructions is still weak and it is hard to solve practical tasks. Given the limited diversity of text instructions in the provided contractor data, we don’t anticipate the model to possess any significant level of generalization with regard to language instructions. To further verify this point, one needs to collect more diverse and higher-quality text-demonstration pairs data. Anyway, this experimental result still indicates the possibility of optimizing the upstream instruction generator by leveraging the pre-trained decoder. This creates possibilities for developing more interesting applications on GROOT. Additional discussions on text-conditioning are beyond the scope of this paper, and we will leave them for future work.

Reviewer comment (Writing):
Nit: confusing notation – changing from $p_\theta$ to $p_\psi$ in the KL term.

Author response:
Thank you for raising this question. We apologize for the confusion caused. It was a typo on our part. Our intention was to clarify that the two video encoders do not share the same set of parameters. In reality, we should use $p_\theta(g | s_{0:t})$ instead of $p_\psi(g | s_{0:t})$. We have corrected this in the paper.

Reviewer comment (Experiments):
It’s interesting that conditioning on multiple videos concatenated together can drive the agent to accomplish multiple tasks simultaneously, and the embedding space seems to be qualitatively meaningful. Do the authors think there would also be a benefit to constructing synthetic clips concatenating multiple tasks and training on them as well? Even if the source video settings look very different (potentially making goal-space learning harder), it might be an interesting way of generating large amounts of interesting synthetic, multi-task data.

Author response:
Thank you for proposing the very interesting idea of training GROOT directly on multi-task concatenated videos to enhance its ability to perform multiple tasks simultaneously. We believe this idea is reasonable and natural. Although the original GROOT demonstrated some ability to perform multiple tasks on concatenated videos, its performance on individual task decreased. We attribute this to inconsistencies between the distribution of training data and the distribution of test data. GROOT trained on concatenated videos would likely alleviate this issue. However, due to resource and workload constraints, we will leave this intriguing idea for future work. Thank you again for your valuable suggestion.