Paper: CLIP-Guided Reinforcement Learning for Open-Vocabulary Tasks (ICLR-2024)

Reviewer comment (Novelty):
My main concern is that this method makes more assumptions than the prior work it is compared to. Specifically, this method assumes the task involves navigating to a target object that is specified in the instruction. An example of an instruction where this method would not work as well is "build a tower" (since there is no target object to move toward). Notably, MineCLIP, STEVE-1, and Cai et al do not make this assumption.

Author response:
Dear reviewer, we are happy to receive your response and appreciate the opportunity to further discuss and address your concern.

> While it's fine to propose a method with more assumptions and more limited applicability, the paper should make those assumptions clear. The paper currently claims that the proposed method solves open-vocabulary tasks, implying the method is applicable to all tasks that can be specified with language.

In the original version of our paper, we claimed "By open-vocabulary task, we mean that the agent is instructed to interact with diverse objects beyond the training scope." at the beginning of Section 2, by which we limit the scope of tasks that our method is designed to solve. We acknowledge, however, that this description may not be precise enough. Therefore, based on the comments from reviewers, we append an additional statement to provide a more restrictive claim on our assumption and application: **"More specifically, we focus on object-centric tasks and the open-vocabulary ability over target objects."**, in the latest revision. Thank you for bringing this to our attention, and we hope this clarification aligns with the paper's objectives more accurately.

> Additionally, the proposed method only outperforms much simpler baselines (one-hot and CLIP embedding conditioning) in the hunting domain.

Firstly, we would like to highlight that our proposed method outperforms baselines not only in the hunt domain but also in the harvest domain. While the learning curves of the three models on **training instructions** appear comparable, as shown in Figure 8(a), the gap between our method and the baseline is significant on **unseen instructions** during test, as illustrated in Figure 8(c). This result demonstrates our method's superior ability to generalize to unseen instructions than EmbCLIP, suggesting that our proposed approach of taking 2D confidence map as a policy conditioning representation offers better open-vocabulary ability over target objects compared to relying on language descriptions.

Secondly, we think that using a one-hot vector as the task indicator (One-Hot) and conditioning policy on language (EmbCLIP) are not simple, but are essential instead. COPL, EmbCLIP, and One-Hot share the same network architecture and training procedures, thus enabling a clear and direct comparison of generalization ability between policy conditioning representations from different modalities (vision, language, and vector), excluding the influence of other factors.

> ... this proposed reward is only valid for tasks that involve moving towards a target object while the prior methods work for more general Minecraft tasks (MineClip, STEVE-1, Cai et. al).

We acknowledge the limitation of COPL in the aspect of generality, as we newly added description in Section 2. At last, we would like to offer a comprehensive perspective that combines both theoretical and empirical considerations here.

**MineCLIP.** Although theoretically, MineCLIP can align video clips with a wide range of language descriptions, besides object-centric tasks, previous work [1,2] and our experiments demonstrate its practical limitations. We find that in practice, MineCLIP still tends to align entities in video clips with those in language descriptions like the original CLIP, instead of utilizing temporal information (otherwise the MineCLIP reward would effectively guide the agent to approach targets in object-centric tasks).

**Imitation learning.** STEVE-1 struggles with accurately differentiating targets, as evidenced by its low performance in the hunt domain. Cai et al. (2023) show limited generalization ability over target objects, as illustrated in Figure 8(b), due to its condition on language input. This aligns with our observations on EmbCLIP. Theoretically, we can attribute these deficiencies to the limitation of the imitation learning dataset, instead of the methods themselves. However, from another perspective, reinforcement learning shows advantages here of being independent of the quality and distribution of the demonstration dataset.

In summary, no method can perfectly handle all tasks and perform zero-shot generalization. Our method could be regarded as a practical compromise that **sacrifices some of MineCLIP's theoretical capabilities of generality but enhances the practical performance for object-centric tasks**, which compose a large portion of tasks in Minecraft.

[1] Cai et al., 2023, Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction.

[2] Yuan et al., 2023, Plan4MC: Skill Reinforcement Learning and Planning for Open-World Minecraft Tasks.

Reviewer comment (Evaluation):
The comparison to imitation learning methods like STEVE-1 and Cai et al should be justified since the proposed method does online RL. Do the imitation learning methods see more or less relevant data for the evaluation tasks? Imitation learning and online RL are different classes of methods so some explanation is needed here.

Author response:
Thank you for your efforts and valuable comments. We would like to address the weaknesses and questions you raised in the review.

> Comparison to imitation learning methods.

In our study, the comparison between COPL and the imitation learning methods, STEVE-1 and Cai et al., is due to the lack of alternative Minecraft foundation models with goal- or instruction-conditioned policies trained via reinforcement learning. We select STEVE-1 and Cai et al. for their capabilities in mastering multiple basic skills in Minecraft, similar to the objectives of our open-vocabulary evaluation.

We acknowledge that comparing an online RL method with imitation learning methods is not a perfect match since the training data is quite different. Therefore, our intention is not to determine which method performs best in learning basic skills in Minecraft. Instead, we just use STEVE-1 and Cai et al. as a reference to understand how COPL performs in this specific domain.

Reviewer comment (Experiments):
Why wasn't EmbCLIP evaluated on unseen instructions in the hunt domain?

Author response:
Thank you for your efforts and valuable comments. We would like to address the weaknesses and questions you raised in the review.

> Why wasn't EmbCLIP evaluated on unseen instructions in the hunt domain?

We do not evaluate EmbCLIP on unseen instructions in the hunt domain due to its limited performance on the training instructions. Since the model does not perform well on tasks it is trained for, we determine that it would not provide meaningful insights to test it on unseen instructions.

Reviewer comment (Novelty):
The main problem with the approach is that it does not seem to be very “general.” This would not be a problem per se (not all ICLR papers should aim at “general” solutions to problems), if not for the fact that the work builds directly on top of MineCLIP, which was aimed at producing a multi-task “general” agent for open-ended Minecraft tasks.

Author response:
As mentioned above, our method could be viewed as a framework applicable to diverse environments, not limited to the specific use case of Minecraft. In Minecraft, we use MineCLIP to realize the first component of our method, "turning a suitable open-vocabulary image segmentation model into a 2d confidence map". We do not intend to build a new fundamental model or method that theoretically surpasses MineCLIP in terms of multimodal ability and generality in Minecraft. Instead, it just serves as a key module specific to Minecraft within our framework to solve the problem we defined in Section 2. In other first-person view and object-centric environments, our framework can adapt by replacing the modified MineCLIP with general models like Grounding DINO + SAM or domain-specific models.

Reviewer comment (Novelty):
The original MineDojo paper attempted to solve all kinds of Minecraft tasks based on their language descriptions (“milk a cow”, “hunt a sheep”, “combat a zombie”, “find a nether portal”, “dig a hole”, “lay a carpet”). For this reason, they proposed a “general” method making use of a MineCLIP encoder, encoding image sequences and language commands, which is not limited to a specific task domain. The MineCLIP encoder can in principle encode image sequences for every Minecraft task, regardless of whether it involves entities or not.

Author response:
Dear reviewer,

Thank you for your response.

> If we expect this embedding to be useful to condition policies for any other task (such as "place a carpet"), we have lost all information. This is what I meant when I said that the proposed method reduces the generality of MineCLIP.

We append an additional statement in Section 2 "Problem Statement" to provide a more restrictive claim on our assumption and application: "More specifically, we focus on object-centric tasks and the open-vocabulary ability over target objects.", in the revision. Given the limited practical performance of MineCLIP reward in our experiments, our method could be regarded as a practical compromise that sacrifices some of MineCLIP's theoretical capabilities of generality but enhances the practical performance for object-centric tasks, which compose a large portion of tasks in Minecraft.

> ... thus better comparisons would have been obtained if starting from such a multitask baseline, adding your focal reward technique on top.

Following your suggestion, we conduct a multi-task experiment based on the Animal Zoo setting introduced in the original MineCLIP paper, including 'milk a cow', 'hunt a cow', 'shear a sheep', and 'hunt a sheep'. As shown in Figure 27, our focal reward outperforms the MineCLIP reward across all tasks. We believe this experiment could serve as a better and more convincing comparison.

We note that the results of the MineCLIP reward are not the same as those in the original paper. Results in the original paper exhibit high success rates in 'hunt a cow' and 'milk a cow' while ours show high success rates in 'milk a cow' and 'shear a sheep'. This disparity suggests that the influence of the fact that we do not use action smoothing, self-imitation learning, and multi-stage training might not be that 'minor' as we mentioned in previous response.

Reviewer comment (Theory):
The “COPL” technique essentially consists of a method for turning a suitable open-vocabulary image segmentation model into a 2D confidence map that helps as a better task-conditioned input for the policy MLP and as a more well-shaped reward model, biased towards looking at entities and getting close to them. This introduces a bias towards entity-focused tasks.

Author response:
If the goal is only to learn tasks in Minecraft, we agree that using traditional reward shaping in the state space of the Minecraft world, such as the 'manual reward' designed in the original MineCLIP paper, is a direct approach. But from our perspective, we wish our proposed methodology applicable in diverse environments, rather than only in Minecraft, for the problem defined in Section 2. As you succinctly summarized, "Essentially, the 'COPL' technique consists of a method for ... getting close to them", the whole framework is not tailored for Minecraft and can also be applied to other similar first-person view and object-centric environments such as robotic navigation. We use Minecraft as a convenient and suitable testbed to validate our method. Therefore, we avoid the use of environment-specific information to guide reinforcement learning.

Reviewer comment (Experiments):
In the experiment section, most results that clearly favor the COPL method belong to the “hunt” domain, which naturally benefits from the COPL reward function’s biases. This could be seen as “overfitting” to hunting-style tasks.

Author response:
In Figure 8(a), the learning curves show the success rates of three models on **training instructions** throughout the training process. The similarity in these curves indicates that all methods have comparable performance on the training tasks. The results demonstrated in Figure 8(c) and (d) include training instructions and **unseen instructions** (highlighted in green). The gap between COPL and EmbCLIP on training instructions ('milk a cow' and 'shear a sheep') is narrow and approximately aligns with the gap observed at the end of the training in Figure 8(a). However, the gap widens on unseen instructions ('harvest water', 'shear a mushroom cow', and 'harvest sand'), suggesting COPL's superior generalization ability.

Specifically, for EmbCLIP, while the text encoder can produce a language embedding for the novel target object, this embedding is essentially out-of-distribution for the policy network. This is because the language embedding space is huge but the policy network is only trained on a limited set, *i.e.*, embeddings of four training target objects. In contrast, COPL's policy network takes a 2D confidence map as input. After our modified MineCLIP converts the unseen target name into a confidence map, the policy network can process it without a generalization gap.

Reviewer comment (Novelty):
If the domains and settings for which this method is useful are so restricted, it is unclear why traditional reward shaping in the Minecraft world’s state space would not be preferable, since it is also not general-purpose but could be strong for specific tasks.

Author response:
If the goal is only to learn tasks in Minecraft, we agree that using traditional reward shaping in the state space of the Minecraft world, such as the 'manual reward' designed in the original MineCLIP paper, is a direct approach. But from our perspective, we wish our proposed methodology applicable in diverse environments, rather than only in Minecraft, for the problem defined in Section 2. As you succinctly summarized, "Essentially, the 'COPL' technique consists of a method for ... getting close to them", the whole framework is not tailored for Minecraft and can also be applied to other similar first-person view and object-centric environments such as robotic navigation. We use Minecraft as a convenient and suitable testbed to validate our method. Therefore, we avoid the use of environment-specific information to guide reinforcement learning.

Reviewer comment (Experiments):
By limiting the CLIP model to a set of pre-determined negative words, isn't the paper limiting the scope and moving away from the actual aim of being open vocabulary?

Author response:
These pre-determined negative words are used to instruct the model on what to ignore, thereby enhancing the accuracy of the 2D confidence map by reducing noise. Concretely, we compute similarities between these negative words and image patches, in addition to similarities between the target object name and image patches. By applying a softmax function to these similarities on each patch, we suppress the activation of the target object on non-target patches, especially where objects from the negative word list are likely present. This results in a cleaner, low-noise confidence map.  

Given the principle mentioned above, we construct the negative word list containing objects that frequently appear in Minecraft for effective denoising. Note that while we limit the negative words, we do not restrict the range of target objects. So in principle, our modified MineCLIP can generate the confidence map for any target object and does not move away from the aim of being open-vocabulary.

Reviewer comment (Experiments):
To extend the previous question, is it possible to perform a similar analysis, but instead of negative words, use the entire vocabulary?

Author response:
If 'the entire vocabulary' refers to all object names in Minecraft, we believe it will work as well because our negative is equivalent to the most frequent part of these objects and is proven to be effective. If 'the entire vocabulary' means the general vocabulary that CLIP can understand, we suppose that it would be less effective. There are multiple reasons. For example, synonyms for the target object will suppress its activation on the confidence map because the synonyms also have high similarities on the patches where the target object presents. Additionally, words with different levels of granularity will complicate the segmentation results. For instance, a human might be identified as 'human' as a whole but also as 'head', 'arms', etc., separately.  

Therefore, a pre-determined, domain-specific word list is essential for open-vocabulary tasks. This is also evidenced in open-vocabulary learning studies, such as open-vocabulary segmentation [1,2,3], where benchmark datasets would provide a pre-determined class set for segmentation evaluation. We hope this could also address your concerns about negative words raised in Weaknesses.  

[1] Ding et al., 2022, Open-Vocabulary Panoptic Segmentation with MaskCLIP.  

[2] Zhou et al., 2022, Extract Free Dense Labels from CLIP.  

[3] Liang et al., 2023, Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP.

Reviewer comment (Experiments):
Would it make sense to keep the objects fixed and change the intended behavior to a similar but nuanced variant of the behavior? Again, I have limited knowledge of Minecraft as a game and have limited knowledge about possible behaviors. However, it seems very logical to me to test open vocabulary capabilities where the agent might 'catch' an animal rather than 'kill' an animal where catching is out of distribution. These behaviors are not very different from training behavior as 'exploring the world' is to it.

Author response:
We appreciate this insightful suggestion and conduct relevant tests to explore our model's ability to adapt to similar but nuanced behavior patterns. We test the hunt domain model on the 'milk a cow' task, where the object 'cow' is familiar, but the required behavior pattern is slightly different. In detail, hunt tasks require 'attack' action to interact but 'milk a cow' requires 'use' action. The hunt model achieves a 43(±16)% success rate, lower than that of the harvest model trained on this task but significantly higher than zero, suggesting that COPL can partially generalize to a nuanced variant of the training behavior pattern.  

Another relevant example is the test task 'harvest sand' in the harvest domain, where both object and behavior are out of the training distribution. As we introduced at the end of Appendix B.3.3, 'harvest sand' requires a slightly different behavior from the training task, and at the same time, 'sand' is a novel target object. As demonstrated in Figure 8(c), COPL outperforms the baseline EmbCLIP on this task.  

However, it is important to note that this generalization may stem from the stochastic or noisy nature of the policy output and does not represent that COPL has a true open-vocabulary ability over verbs.

Reviewer comment (Novelty):
The core contribution of the paper is an intrinsics reward for minecraft with a lot of limitations. The reward seems to be specifically tailored for minecraft's first-person-view setting, and specifically towards tasks that involve approaching an object.

Author response:
> The core contribution of the paper is an intrinsics reward for minecraft with a lot of limitations ...

We acknowledge that our method is primarily designed for first-person view and object-centric tasks. But we want to clarify that this setting is not exclusive to Minecraft, but contains a wide range of realistic and general applications. For example, embodied AI research domains, such as autonomous driving, navigation, and robotic manipulation, often employ first-person views and/or focus on object-centric interactions. This setting better mirrors the complexity of real-world challenges compared to canonical RL environments, where our method may not be suitable.

Additionally, we would like to emphasize that our contribution also includes a novel policy conditioning representation, namely the 2D confidence map. Using a 2D confidence map to replace language input substantially improves the generalization ability of learned policy, as demonstrated in Section 4.2. Although currently restricted to object-centric tasks, it offers some insights about grounding language into vision to facilitate multimodal policy learning.

We hope this elaboration clarifies the generality and contributions of our work, while also acknowledging and appreciating the limitations you have pointed out.

Reviewer comment (Evaluation):
The tasks have to be object-centric, and the generalization is mostly object level.

Author response:
> The core contribution of the paper is an intrinsics reward for minecraft with a lot of limitations ...

We acknowledge that our method is primarily designed for first-person view and object-centric tasks. But we want to clarify that this setting is not exclusive to Minecraft, but contains a wide range of realistic and general applications. For example, embodied AI research domains, such as autonomous driving, navigation, and robotic manipulation, often employ first-person views and/or focus on object-centric interactions. This setting better mirrors the complexity of real-world challenges compared to canonical RL environments, where our method may not be suitable.

Additionally, we would like to emphasize that our contribution also includes a novel policy conditioning representation, namely the 2D confidence map. Using a 2D confidence map to replace language input substantially improves the generalization ability of learned policy, as demonstrated in Section 4.2. Although currently restricted to object-centric tasks, it offers some insights about grounding language into vision to facilitate multimodal policy learning.

We hope this elaboration clarifies the generality and contributions of our work, while also acknowledging and appreciating the limitations you have pointed out.

Reviewer comment (Evaluation):
While the promise of this paper / minecraft itself is about open-vocabulary, the presented method is limited to approaching objects in FPV setting. This is also reflected in the evaluation, where the tasks covered are not close to open-world. The authors should tone down their claim about open-world.

Author response:
> The core contribution of the paper is an intrinsics reward for minecraft with a lot of limitations ...

We acknowledge that our method is primarily designed for first-person view and object-centric tasks. But we want to clarify that this setting is not exclusive to Minecraft, but contains a wide range of realistic and general applications. For example, embodied AI research domains, such as autonomous driving, navigation, and robotic manipulation, often employ first-person views and/or focus on object-centric interactions. This setting better mirrors the complexity of real-world challenges compared to canonical RL environments, where our method may not be suitable.

Additionally, we would like to emphasize that our contribution also includes a novel policy conditioning representation, namely the 2D confidence map. Using a 2D confidence map to replace language input substantially improves the generalization ability of learned policy, as demonstrated in Section 4.2. Although currently restricted to object-centric tasks, it offers some insights about grounding language into vision to facilitate multimodal policy learning.

We hope this elaboration clarifies the generality and contributions of our work, while also acknowledging and appreciating the limitations you have pointed out.

Reviewer comment (Novelty):
The proposed open-vocab segmentation seems to be very similar to previous methods like [1]. Even if this is disregarded, if the reward is already tailored for minecraft (first-person view + task is mainly approach object), one may well use some open-vocabulary detection model tailored for the few tasks the paper benchmarks. Once one detects the objects from text, the reward the authors propose seems an obvious thing to do.

Author response:
Dear reviewer, we appreciate the opportunity to further discuss and respond to your point about the use of open-vocabulary detection models in Minecraft or other first-person-view (FPV) games. 

**Minecraft**. Firstly, we would like to clarify that currently, there is not an existing open-vocabulary detection model tailored for Minecraft. An alternative approach is to use some off-the-shelf open-vocabulary object detection models directly. However, as detailed in our newly appended Appendix A.4, due to the significant domain gap between visuals in Minecraft and the real world, direct application of these models to Minecraft results in inaccurate detection. In other words, they require adaptation to the Minecraft domain, implying additional costs associated with labeled data collection and fine-tuning. In contrast, our modification on MineCLIP does not demand further fine-tuning, allowing for immediate and effective implementation.

**Other FPV games.** Similar to Minecraft, other FPV games also suffer from the domain gap between their visual styles and the real world more or less. As mentioned previously, if we want to adapt an off-the-shelf open-vocabulary detection model to a specific game domain, a labeled game dataset is required. Annotating object bounding boxes and names demands extensive human labor. However, the wealth of game videos on the Internet, such as those on YouTube, presents an alternative approach. These videos often come with subtitles, providing natural labels that can be utilized for training. Therefore, compared to adapting an object detection model, training a CLIP-like vision-language model (VLM), such as CLIP [1] and CLIP4Clip [2], on this massive data would be much more labor-saving and automated. Most importantly, MineCLIP has already demonstrated this route is feasible. Once a domain-specific CLIP-like model is obtained for a game, our method could be then implemented based on this VLM to guide the agent to learn basic skills in this game.

In summary, compared to tailored open-vocabulary detection models, our method could be nested in a general pipeline with VLM learning, which requires less labor and is more automated, for FPV games. We hope the discussion above addresses your concern. If you still have any additional questions or comments, please do not hesitate to let us know.

[1] Radford et al., 2021, Learning Transferable Visual Models From Natural Language Supervision.

[2] Luo et al., 2021, CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval.

Reviewer comment (Novelty):
I am not exactly sure whether the authors claim the architecture in figure 2 to be a main contribution. If so, the authors should probably discuss previous approaches and how their approach is different, either when mentioning MaskCLIP for the first time or in related work.

Author response:
> I am not exactly sure whether the authors claim the architecture in figure 2 to be a main contribution.

The architecture illustrated in Figure 2 is not the main contribution of our study but rather a key module within our overall framework. As introduced in Section 3.1, it is adapted from MaskCLIP. The differences between our modified MineCLIP and MaskCLIP are twofold: firstly, MineCLIP includes an additional temporal transformer in the vision pathway; secondly, our modified MineCLIP outputs a confidence map for the target object instead of a semantic segmentation result.

Reviewer comment (Theory):
From my understanding, for this reward to work properly, the object has to be already in FOV, correct?

Author response:
> From my understanding, for this reward to work properly, the object has to be already in FOV, correct?

Getting a non-zero focal reward indeed requires the target object to be in the agent's field of view. When the target object is not present in the FOV, resulting in a confidence map with no high-probability patches, the focal reward is zero. This characteristic would prompt the agent to look around immediately after spawning in the Minecraft world to find the target object. In our experiments, target objects often appear in the agent's observable range, which means that the agent can see the target object when facing the direction of the target object.

If we consider a more challenging setting where the target object is located out of the agent's observable range at the spawning point, an additional exploration algorithm should be integrated to encourage the agent to move around. For example, incorporating novelty-induced exploration.

Reviewer comment (Experiments):
The approach does have some hand-crafted components that are specific to this domain. For example, the Gaussian kernel in the focal reward relies on the fact that interaction in Minecraft occurs "when the cursor in the center of the agent view aligns with the target" and also guides the agent to focus on a single target rather than multiple. How was this kernel chosen and how sensitive is the performance of the method to the specific choice of kernel? Another example is the negative word list; while effective for denoising, it is determined in a domain-specific fashion, so it is unclear if the benefit of this procedure would be helpful for other domains.

Author response:
We would like to clarify that constructing a negative word list is inherently domain-specific. For instance, in another domain, indoor navigation tasks, the list should be constructed to contain indoor objects rather than outdoor ones. Although the approach may vary with each domain, the concept of building a task-relevant negative word list to denoise and enhance segmentation effectiveness is universally applicable.  

The Gaussian kernel in our method is constructed based on the simple principle that values should be higher near the center and decrease with distance. $\sigma=(H/3,W/3)$ is an intuitive choice, aiming to maintain high contrast between the center and the edge, as well as between the edge and areas outside the field of view.  

To quantitatively evaluate this intuition and answer the second question, we conduct additional experiments to test the sensitivity of our methods to Gaussian kernels with different variances and add the results to Appendix B.4. The experimental settings are the same as those in Figure 6(c) and (d). As illustrated in Figure 21, $\sigma=(H/3,W/3)$ outperforms the other two choices. We suppose that the Gaussian kernel with $\sigma=(H/2,W/2)$ is too wide to provide sufficient contrast between the center and the edge, while the Gaussian kernel with $\sigma=(H/5,W/5)$ is too narrow to provide sufficient contrast between the edge and areas outside the field of view.

Reviewer comment (Experiments):
The choice of adding the unified 2D confidence map to the policy input is an interesting way to get some invariance across objects. But removing the natural language input constrains the policies to be single task instead of multi-task policies. What is the advantage of removing natural language? One rationale might be that unseen objects which are OOD for the policy do not have to be encoded by the text encoder--but these unseen objects are already being encoded by the visual encoder, so it is unclear if this is the reason. Was the choice of not including text as a policy input ablated?

Author response:
We do not include the natural language instruction as input to the policy primarily because the instruction cannot offer additional useful information beyond what is already provided by the 2D confidence map in our Minecraft experiments. The information in the language instruction, besides the target object, often consists of verbs like 'hunt', 'harvest', or 'collect'. However, such actionable information does not significantly aid in task completion. On the one hand, the action patterns required to complete tasks in Minecraft are relatively simple, *i.e.*, move towards the target object and take 'attack' or 'use' action. On the other hand, the mapping between verbs and action patterns is not one-to-one. For example, an instruction with the verb 'collect' may require the agent to either 'attack' or 'use' depending on the type of the target object. Therefore, we believe that language instruction cannot provide additional useful information for the agent's policy.  

We acknowledge that in other domains like robotic manipulation, language instructions often contain essential information, such as verbs 'pick' and 'push' indicating totally different behavior patterns. Under this circumstance, taking as input the language instruction will be necessary.

Reviewer comment (Experiments):
How sensitive is $\lambda$ across multiple task families?

Author response:
In our experiments, we observe that the average focal reward per step consistently falls into a range of 0.1 to 0.3 across all tasks due to the similar state spaces (all tasks are evaluated in Minecraft). The focal rewards' uniformity in order of magnitude across task families indicates the robustness of our $\lambda$ selection, although it is derived from 'hunt a cow' task and 'hunt a sheep' task. To validate this, we conduct additional experiments on two tasks from the harvest task family, 'milk a cow' and 'shear a sheep', and add the results to Appendix B.4. As shown in Figure 20(a) and (b), $\lambda=5$ still outperforms other values on these two tasks.  

While we acknowledge that $\lambda=5$ may not be the exact optimal value for every individual task, it is a reasonably near-optimal choice (*e.g.* on the same order of magnitude as the optimal value), serving as a broadly effective parameter for our multi-task training.

Reviewer comment (Reproducibility):
How was the Gaussian kernel constructed?

Author response:
The Gaussian kernel in our method is constructed based on the simple principle that values should be higher near the center and decrease with distance. $\sigma=(H/3,W/3)$ is an intuitive choice, aiming to maintain high contrast between the center and the edge, as well as between the edge and areas outside the field of view.  

To quantitatively evaluate this intuition and answer the second question, we conduct additional experiments to test the sensitivity of our methods to Gaussian kernels with different variances and add the results to Appendix B.4. The experimental settings are the same as those in Figure 6(c) and (d). As illustrated in Figure 21, $\sigma=(H/3,W/3)$ outperforms the other two choices. We suppose that the Gaussian kernel with $\sigma=(H/2,W/2)$ is too wide to provide sufficient contrast between the center and the edge, while the Gaussian kernel with $\sigma=(H/5,W/5)$ is too narrow to provide sufficient contrast between the edge and areas outside the field of view.

Reviewer comment (Novelty):
Given that the language instructions are fairly simple, what is the rationale for using an LLM to find the target object?

Author response:
The use of an LLM is intended to address the challenge where the name of the target object is not explicitly mentioned in the language instruction. For example, consider the instruction 'obtain wool with shears'. Here, 'wool' is the target item to acquire, but the actual target object for the agent to approach and interact with is 'sheep', which does not appear in the instruction. It is hard to design a programmatical method to extract such an implicit target object from the instruction. Consequently, we resort to LLMs, which understand the underlying relationship between the target item (wool) and the target object (sheep). More examples are available in Appendix A.1.