Paper: Language Reward Modulation for Pretraining Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
It is not clear to me what is the main contribution of the proposed method. Many, if not most, of the components in the proposed method come from prior works, such as R3M, Plan2Explore, vision-language-embedding matching and so on. Meanwhile, the idea of conditioning the reward function on instructions/goals is also not novel, see the following Point 2.

Author response:
**Primary Contribution**

The primary contribution of this work is to propose a possible route towards pretraining a single generalist RL agent with limited human supervision that can efficiently adapt to many new tasks. Toward this goal we select a variety of downstream finetuning tasks for evaluation and do not include any of these tasks in our pretraining setup nor do we enforce any very strict requirements on their connection to the pretraining tasks. R3M has been used as a visual encoder to accelerate behavior learning by initializing with generalizable representations, however, to our knowledge has not been used as a reward function. Furthermore, instead of using a VLM to specify a task reward for a single language instruction or image goal as is done in [2][3], we demonstrate experimentally that by using a diverse set of language prompts with a frozen VLM, LAMP can be pretrained to do a wide array of tasks even more effectively than with the most widely-used exploration algorithms (Plan2Explore, RND). Our pretraining design is a divergence from existing approaches both in our inclusion of zero-shot VLM rewards from representation learning models and the way in which we generate them with diverse language prompting styles with loose relevance to the affordances in the pretraining scene. We also demonstrate a synergy between these pretrained components and fully unsupervised methods which to our knowledge has not been experimentally demonstrated. Lastly, we conduct a systematic evaluation of which language promoting techniques are best suited for the pretraining setting, which we anticipate to be of interest to the community as VLMs grow in popularity.

Reviewer comment (Novelty):
It seems missing the connection of the proposed method with goal-conditioned RL and distinction from them. For example, how is prompting VLM with different language instructions different from inputting different goal states to the goal-conditioned reward function?

Author response:
**Connection to Goal-Conditioned RL**: Our method is related to goal-conditioned RL in the sense that we learn a language-conditioned agent and language instructions can be considered semantic goals. However, the conventional goal-conditioned RL setup does not address the desiderata of scalable behavior pretraining. In general, this setting requires human supervision to select interesting goals for the agent to reach and actually achieving those goals often remains an exploration challenge. A contribution of LAMP is in identifying a compatibility of the relative ease of enumerating exploratory language goals compared to image goals and the constraints on human involvement for open-ended pretraining.

Reviewer comment (Experiments):
The proposed pre-training stage seems computationally expensive, requiring rollouts, language and visual embeddings, calling the R3M score predictors, and so on, before updating the RL components.

Author response:
**Pretraining Desiderata and Computational Cost**

There have been considerable efforts to leverage VLMs such as [4] as a task reward. In fact, works such as [5] select datasets and architectures towards producing general-purpose reward functions. However, we argue in this work that due to issues of reward exploitation and noisy models that these LRF’s are ill-suited for directly learning downstream tasks. We instead argue that the pretraining phase is a more favorable context, which we refer to as “lower-stakes”, as we are reasonably less concerned with the sample-inefficiency of a “one-time cost” that requires limited human supervision than for a recurring cost we incur each time we desire to quickly learn a new skill. This is corroborated by [7]. Moreover, the differences in desiderata between pretraining and finetuning make the admittedly somewhat computationally intensive nature of LAMP more palatable. The paradigm of computationally expensive, large-scale pretraining in service of multi-purpose few-shot or task-specific finetuning characterizes much of modern machine learning [6].

Reviewer comment (Evaluation):
Considering the relatively wide error bars in Fig. 5, the performance gain of the proposed method LAMP may not be clear when comparing with the baseline P2E.

Author response:
Regarding the 4th concern in the original response - LAMP consistently outperforms the baselines across the very different downstream tasks evaluated. The finetuning curve can be high variance, however, the trend holds averaged over seeds and across different tasks and ablations. We have added many more experiments that evidence this consistent trend in Section 6.2.

Reviewer comment (Experiments):
Figure 7 may not convincingly support the claim that the proposed method "is not inherently reliant on a particular VLM", since the authors only show the results of ZeST besides the main choice R3M.

Author response:
Regarding the 5th concern in the original response, we have adjusted the language in the paper. We have removed the claim that "LAMP is not inherently reliant on a particular VLM" and instead clarify that LAMP can work well with CLIP rewards in addition to R3M rewards. Please see Section 6.3 of the updated paper.

Reviewer comment (Theory):
Why is the pretraining stage lower-stakes? Do you have any reference or justification?

Author response:
**Pretraining Desiderata and Computational Cost**

There have been considerable efforts to leverage VLMs such as [4] as a task reward. In fact, works such as [5] select datasets and architectures towards producing general-purpose reward functions. However, we argue in this work that due to issues of reward exploitation and noisy models that these LRF’s are ill-suited for directly learning downstream tasks. We instead argue that the pretraining phase is a more favorable context, which we refer to as “lower-stakes”, as we are reasonably less concerned with the sample-inefficiency of a “one-time cost” that requires limited human supervision than for a recurring cost we incur each time we desire to quickly learn a new skill. This is corroborated by [7]. Moreover, the differences in desiderata between pretraining and finetuning make the admittedly somewhat computationally intensive nature of LAMP more palatable. The paradigm of computationally expensive, large-scale pretraining in service of multi-purpose few-shot or task-specific finetuning characterizes much of modern machine learning [6].

Reviewer comment (Evaluation):
Is there any intuition why the $r^{VLM}$ is combined with the intrinsic rewards from Plan2Explore, which is itself a novelty-seeking score? Is there any intuition why in Fig. 7 the performance degrades quite significantly when using $r^{VLM}$ alone to pre-train the agent?

Author response:
Regarding Question 3, we take motivation from findings in [8] that maximizing rewards from pretrained models can be a challenging optimization that suffers from reward exploitation and convergence to local minima. The paper states "In the reinforcement learning setting, an additional issue is that solely optimizing a dense reward such as VIPER can lead to early convergence to local optima." and "In the reinforcement learning literature, the entropy bonus over input trajectories corresponds to an exploration term that encourages the agent to explore and inhabit a diverse distribution of sequences within the regions of high probability under the video prediction model... VIPER is agnostic to the choice of exploration reward and in this paper we opt for Plan2Explore and RND." We similarly found empirically in our setting that adding the intrinsic novelty bonus enables the RL agent to explore more modes of a noisy reward function with many local minima and thus leads to better performance.

Reviewer comment (Writing):
The $R_t$ just below the title of Section 3 seems missing an expectation symbol?

Author response:
Regarding Question 4, thank you for the catch regarding the expectation notation for the RL objective - see the revised paper with this fixed.

Reviewer comment (Experiments):
Are there any requirements on the downstream tasks so that the pretraining pipeline in Section 4.2 remains helpful?

Author response:
1. Much of the allure of general purpose pretrained models is that they can be used for many downstream tasks. Our pretraining setup is designed for this in that we do not assume knowledge of the diverse downstream tasks in pretraining the agent - they are completely unseen at finetuning time. As $n$ grows, the "new baseline" proposed will become increasingly more weak as the pretraining budget per task shrinks. However, since for our method the agent is only pretrained once, the sample complexity does not scale as poorly with the number of tasks. This is because, as we demonstrate in the paper, learning each of 5 very semantically and behaviorally diverse tasks is more sample-efficient from a pretrained LAMP agent. The agent performs a closing behavior, a valve turning behavior, fine-grained picking behaviors, and a pushing behavior all benefitting from the same pretrained model. We additionally note that for many robotics tasks, seemingly small modifications to the task can harm the success rate of a pretrained agent [9]. If instead of training an agent from scratch each time, we can more efficiently learn by warm-starting training from a pretrained agent, we further realize the benefits of task-agnostic finetuning.

Reviewer comment (Experiments):
How sensitive is the proposed method to the $\alpha$ parameter in Eq. 2?

Author response:
**Desired Results in Appendix**

We were similarly interested to understand the impact of the reward weighting term. We had already provided these results in the Appendix. Please refer to Figure 11 in Section D.2 of the Supplementary Material. We find that LAMP works best with our choice of alpha but still exhibits very strong performance for an alpha value of 0 corresponding to training only with VLM rewards and no Plan2Explore bonus. We think this is quite an encouraging result and hope this addresses the reviewers concerns.
We were additionally interested in how another novelty-seeking algorithm would perform. We direct the reviewer to an additional widely-used baseline, Random Network Distillation (RND) [1] in the Appendix (Figure 13, Section D.3.1). LAMP is able to outperform this baseline on all of the tasks evaluated.

Reviewer comment (Evaluation):
In Figure 6 (right), why does LAMP Prompt 6 (w/o P2E) perform so well, close to LAMP under Prompt Style 2 (even with P2E)? This is a bit counter-intuitive to me since Prompt 6 is "Snippets from Shakespeare".

Author response:
For our thoughts on Question 7, see the end of the second to last paragraph of Section 6.1 on language prompting. We state "we hypothesize that both exploration coverage and exploration efficiency during pretraining play an important role. By separately increasing exploration coverage through Plan2Explore, the quality of the VLM rewards becomes more important for focusing the auxiliary exploration objective on useful exploratory behaviors. Thus, LAMP Prompt 2, which incorporates Plan2Explore, is trained on higher-quality, more relevant rewards, and can explore more efficiently during pretraining, and therefore enables more effective finetuning."

Reviewer comment (Experiments):
Is there an ablation study on the novelty-seeking RL pretraining methods. For example, what if we change Plan2Explore to other methods?

Author response:
**Desired Results in Appendix**

We were similarly interested to understand the impact of the reward weighting term. We had already provided these results in the Appendix. Please refer to Figure 11 in Section D.2 of the Supplementary Material. We find that LAMP works best with our choice of alpha but still exhibits very strong performance for an alpha value of 0 corresponding to training only with VLM rewards and no Plan2Explore bonus. We think this is quite an encouraging result and hope this addresses the reviewers concerns.
We were additionally interested in how another novelty-seeking algorithm would perform. We direct the reviewer to an additional widely-used baseline, Random Network Distillation (RND) [1] in the Appendix (Figure 13, Section D.3.1). LAMP is able to outperform this baseline on all of the tasks evaluated.

Reviewer comment (Experiments):
I have some doubts about the usefulness of the core of the proposed method for the realistic setting. It seems that the best reported results are obtained when the semantic tasks for exploration were based on the task that was later used for fine-tuning (prompt style 2 uses reformulations of that task). How realistic is the assumption of knowing the target task at the pre-training stage? Does it mean that the method would require a new pre-training stage when the task changes? In that case, why not use the exploration budget instead for extra data collection for the task of interest? Also, the authors used the scripted reward based on the environment state for the fine tuning on the task of interest. Why not continue using the learnt reward model in this case? In a real world scenario this would be the only way to obtain the rewards.

Author response:
**Important Clarification of Pretraining Setting and Contribution**

It appears that Reviewer PM9W may have missed that the pretraining phase is fully finetuning task-agnostic and a “one-time cost.” During the pretraining phase, the agent optimizes different rewards and encounters many different but randomly-sampled tasks. The analysis we perform during pretraining examines various prompting styles and their relevance to the pretraining tasks, which we assume knowledge of for certain prompting styles, and not the finetuning tasks, which we assume no knowledge of. Prompt Style 2 is not based on the finetuning tasks but the pretraining tasks. VLM-guided RL pretraining is only done once, but the same, single pretrained agent is able to achieve the improved finetuning results on all the tasks evaluated. The 5 finetuning evaluation tasks are not included in the pretraining phase. This setup was intentional as this line of work is in pursuit of a single, general-purpose pretrained RL agent that can adapt to arbitrary new tasks with better sample-efficiency. We expect this important clarification to change your evaluation of the contribution as well address your questions. Please inform us if you have further questions.

Using LAMP directly as a task reward would not enable the RL agents to solve any of the tasks. As we evidence in Figure 3, and motivate in the introduction, rewards from current pretrained VLMs are far too noisy and susceptible to reward exploitation to be usable with online RL. In fact, this is central to our message - current VLMs are ill-suited for this purpose in that they lack the required precision to supervise challenging robot manipulation tasks, however, they do possess certain properties that render them very useful for pretraining RL agents with limited supervision. Scripted finetuning task rewards are simply a means to provide a consistent evaluation of the adaptation capabilities of the various pretrained agents. Our contribution is centered on proposing an RL agent pretraining setup.

Reviewer comment (Experiments):
The paper only considers two baselines for the proposed method: training from scratch or using an exploration objective. There is a large body of literature on exploration for RL pre-training, so it would be nice to add other baselines. In particular, methods that use semantic exploration in RL with the help of pre-trained VLMs are the most relevant. See for example: Semantic Exploration from Language Abstractions and Pretrained Representations. Allison Tam, Neil Rabinowitz, Andrew Lampinen, Nicholas A. Roy, Stephanie Chan, DJ Strouse, Jane Wang, Andrea Banino, Felix Hill. NeurIPS 2022.

Author response:
We thank the reviewer for mentioning an additional baseline. While this baseline is relevant and worth citation, it makes additional assumptions on the environment - in this case access to an oracle language captioner which provides perfect natural langue descriptions of the current visual scene. We direct the reviewer to an additional widely-used baseline with more similar environment assumptions to our own in Random Network Distillation (RND), which is also baselined against in the mentioned work. We provide results in the Appendix (Figure 13, Section D.3.1). LAMP is able to outperform this baseline on all of the tasks evaluated. 

[1] Exploration by Random Network Distillation (Burda et al 2018)

Reviewer comment (Experiments):
It seems that the proposed method includes the component of semantic exploration with only 0.1 weight and the rest is based on the prior work Plan2Explore. I am wondering how sensitive the method is to the choice of this weight? If the proposed objective is used without Plan2Explore, does it still outperform the two baselines?

Author response:
**Desired Results in Appendix**

We were similarly interested to understand the impact of the reward weighting term. We had already provided these results in the Appendix. Please refer to Figure 11 in Section D.2 of the Supplementary Material. We find that LAMP works best with our choice of alpha but still exhibits very strong performance for an alpha value of 0 corresponding to training only with VLM rewards and no Plan2Explore bonus. We think this is quite an encouraging result and hope this addresses the reviewers concerns.

---

R3 -> R3: We have added additional experiments we thought to make the reviewer aware of.

We have added a new Reward Weighting Ablation section to the main text as well as added the ablation on all 5 of the evaluation tasks as opposed to just the Pick Up Cup task from the original manuscript in Section 6.2 in an effort to further enhance clarity. We arrive at a similar conclusion - LAMP is robust to this weighting and performs somewhat competitively even with an alpha value of 0 corresponding to only VLM rewards. However, the synergy of the Plan2Explore and VLM rewards leads to the optimal performance.

We have similarly added a new VLM Ablation Section to the main text and evaluated on all 5 of the evaluation tasks as opposed to just Pick up Cup from the original manuscript in Section 6.3.

Reviewer comment (Experiments):
Why was it necessary to do scene augmentations with the same textures/images that were used for pre-training a VLM (Ego4D)? Could other (independent of pre-training) scene augmentations be used?

Author response:
**Ego4D Texture Overlay** 

We used the Ego4D textures during pretraining to increase the difficulty of the control task with more complex visuals to more closely approximate the real world. These textures help to narrow the domain gap between the robot environment and the datasets used to pretrain the VLMs. We adopt the same texture randomization scheme for all pretraining methods we evaluate for fair comparison.

Reviewer comment (Experiments):
**The weight of plan2explore rewards.** The authors use a weight of 0.9 for plan2explore rewards, and only a weight of 0.1 for their proposed rewards. Would the weight impact much? It could be interesting to see more ablation results for this parameter.

Author response:
**Desired Results in Appendix**

We were similarly interested to understand the impact of the reward weighting term. We had already provided these results in the Appendix. Please refer to Figure 11 in Section D.2 of the Supplementary Material. We find that LAMP works best with our choice of alpha but still exhibits very strong performance for an alpha value of 0 corresponding to training only with VLM rewards and no Plan2Explore bonus. We think this is quite an encouraging result and hope this addresses the reviewers concerns.

Reviewer comment (Experiments):
**Lack of baseline**. I think VIPER [1] and VIP [2] could also serve as baselines. It would be good to see more baseline results in Figure 5.

Author response:
**Other Pretrained Models** 

VIPER is not a VLM in that it is not language-conditioned. Also our focus is on leveraging pretrained VLMs with strong generalization capabilities by providing diverse prompting to generate diverse rewards. VIPER is only shown to work as a reward function on mostly in-domain data. VIP, although it demonstrates some level of domain generalization, is also not a VLM, but rather relies on goal image alignment to compute a reward. Using such a method in a pretraining scheme quite different from that to which we propose would possibly lead to interesting results, however, would require possessing many interesting and diverse exploration goals a priori, which is a strong assumption of open-ended pretraining. Central to our contribution is that diverse language is a uniquely flexible means of generating diverse rewards with relatively little human supervision. In general, the message of our work is that providing diverse prompting with pretrained VLMs to generate rewards can serve as an effective pretraining scheme, not that R3M is the only particular pretrained model that can work.

Reviewer comment (Experiments):
**No results about pre-training stages.** What is the success rate of directly applying rewards of LAMP for in-domain task learning?

Author response:
**LAMP as a task-reward**

To answer simply, the success rate when using LAMP as a task reward on all of the tasks is 0%. As we evidence in Figure 3, and motivate in the introduction, rewards from current pretrained VLMs are far too noisy and susceptible to reward exploitation to be usable with online RL. In fact, this is central to our message - current VLMs are ill-suited for this purpose in that they lack the required precision to supervise challenging robot manipulation tasks, however, they do possess certain properties that render them very useful for pretraining RL agents with limited supervision.

Reviewer comment (Experiments):
As far as I can tell, the main contribution here is the choice of vision-language model, the particular temporal objective (a minor change), and the addition of an exploration reward (Plan2Explore), but if the authors claim that this method is worth further study regardless of the specifics in the paper, they need to be ablated. The exploration reward is ablated (RND shows up as standalone pretraining in Figure 13). The vision-language model is only ablated on a single task (Figure 7). The objective is never ablated, unless we count the comparison to ZeST as a combination, in which case it's unclear that there is a benefit to the proposed scheme.

Author response:
**Ablations**

Regarding the ablations, Reviewer YXuM mentions that we do ablate the choice of exploration reward with inclusion of RND in the Appendix. We agree with this and are glad the reviewer has acknowledged this ablation. Furthermore, they suggest ablating the choice of choice of temporal objective in order to prove that there is “benefit to the proposed scheme.” We do make strong claims that a particular formulation of temporal alignment is uniquely critical for pretraining, rather we propose in the paper that it be used as a language-conditioned pretraining reward and demonstrate the benefits when integrating it in LAMP. We simply employ the most natural alignment objective based on the way the VLM is trained or what has been found to work well in prior work. If the reviewer still disagrees we would appreciate further explanation and suggestions as to what other language-temporal alignment objectives they would like to see. 

To address the last point, the hyperparameter choices for the Masked World Models agent follow the paper which introduced the method [1]. We treat the RL largely as a black box and do not tune RL specific agent hyperparameters as these are orthogonal to our method. We provide the default hyperparameters in the interest of enhancing simple reproducibiilty. The base RL algorithm is kept entirely constant across all baselines (Plan2Explore, RND). The justification for using Masked World Models (MWM) as opposed to a “more standard” RL algorithm such as SAC, PPO, DreamerV3 is that MWM can solve all of the challenging downstream tasks evaluated from scratch with no tuning. We found that this was not the case for the aforementioned more standard algorithms. Lastly, we point the reviewer to our ablation on the balance weighting between exploration reward and VLM reward in Appendix Section D.2 Figure 11 as this hyperparameter is related to our contribution.

[1] Masked World Models for Visual Control (Seo et al 2022)

[2] ShapeNet: An Information Rich 3-D Model Repository (Chang et al 2015)

Reviewer comment (Evaluation):
I'm not sure if the evaluations are separated into in-distribution and out-of-distribution tasks, but it would be worthwhile to understand the performance drop on OOD tasks that were not used in pretraining to further understand the benefit of this pretraining.

Author response:
**Important Clarification of Pretraining Setting and Contribution**

It appears that Reviewer YXuM may have missed that the pretraining phase is fully finetuning task-agnostic and a “one-time cost.” During the pretraining phase, the agent optimizes different rewards and encounters many different but randomly-sampled tasks. The analysis we perform during pretraining examines various prompting styles and their relevance to the pretraining tasks, which we assume knowledge of for certain prompting styles, and not the finetuning tasks, which we assume no knowledge of. The above point addresses “pretraining saves us 50k steps…” and “I wonder if the method is competitive on a fixed-compute budget with single-task training.” VLM-guided RL pretraining is only done once, but the same, single pretrained agent is able to achieve the improved finetuning results on all the tasks evaluated. The 5 finetuning evaluation tasks are not included in the pretraining phase. This setup was intentional as this line of work is in pursuit of a general-purpose pretrained RL agent that can adapt to arbitrary new tasks with better sample-efficiency. Furthermore, to the third point, all 5 of the evaluations are on OOD tasks. The pretraining tasks are randomly sampled from a set of combinatorially many possible tasks using ShapeNet [2] objects. See Figure 8 in Appendix C.2 for examples. We expect this important clarification to change your evaluation of the contribution as well address your questions. Please inform us if we have misinterpreted your concerns or if you have further questions.

Reviewer comment (Experiments):
There are many moving parts in this method (proposed method, Masked World Model policy, Plan2Explore additional reward, architectural choices against baselines, notice the number of hyperparameters in Tables 5, 6, 7, 8 and 9) which are significantly more complex than the standard deep RL setting, and it would be nice to see experiments justifying all of these choices, or at least further argument.

Author response:
**Ablations**

Regarding the ablations, Reviewer YXuM mentions that we do ablate the choice of exploration reward with inclusion of RND in the Appendix. We agree with this and are glad the reviewer has acknowledged this ablation. Furthermore, they suggest ablating the choice of choice of temporal objective in order to prove that there is “benefit to the proposed scheme.” We do make strong claims that a particular formulation of temporal alignment is uniquely critical for pretraining, rather we propose in the paper that it be used as a language-conditioned pretraining reward and demonstrate the benefits when integrating it in LAMP. We simply employ the most natural alignment objective based on the way the VLM is trained or what has been found to work well in prior work. If the reviewer still disagrees we would appreciate further explanation and suggestions as to what other language-temporal alignment objectives they would like to see. 

To address the last point, the hyperparameter choices for the Masked World Models agent follow the paper which introduced the method [1]. We treat the RL largely as a black box and do not tune RL specific agent hyperparameters as these are orthogonal to our method. We provide the default hyperparameters in the interest of enhancing simple reproducibiilty. The base RL algorithm is kept entirely constant across all baselines (Plan2Explore, RND). The justification for using Masked World Models (MWM) as opposed to a “more standard” RL algorithm such as SAC, PPO, DreamerV3 is that MWM can solve all of the challenging downstream tasks evaluated from scratch with no tuning. We found that this was not the case for the aforementioned more standard algorithms. Lastly, we point the reviewer to our ablation on the balance weighting between exploration reward and VLM reward in Appendix Section D.2 Figure 11 as this hyperparameter is related to our contribution.

[1] Masked World Models for Visual Control (Seo et al 2022)

[2] ShapeNet: An Information Rich 3-D Model Repository (Chang et al 2015)

Reviewer comment (Experiments):
Is there a split in the evaluation into seen and unseen tasks? This was the original motivation for the method so I think it is worthwhile to test.

Author response:
**Important Clarification of Pretraining Setting and Contribution**

It appears that Reviewer YXuM may have missed that the pretraining phase is fully finetuning task-agnostic and a “one-time cost.” During the pretraining phase, the agent optimizes different rewards and encounters many different but randomly-sampled tasks. The analysis we perform during pretraining examines various prompting styles and their relevance to the pretraining tasks, which we assume knowledge of for certain prompting styles, and not the finetuning tasks, which we assume no knowledge of. The above point addresses “pretraining saves us 50k steps…” and “I wonder if the method is competitive on a fixed-compute budget with single-task training.” VLM-guided RL pretraining is only done once, but the same, single pretrained agent is able to achieve the improved finetuning results on all the tasks evaluated. The 5 finetuning evaluation tasks are not included in the pretraining phase. This setup was intentional as this line of work is in pursuit of a general-purpose pretrained RL agent that can adapt to arbitrary new tasks with better sample-efficiency. Furthermore, to the third point, all 5 of the evaluations are on OOD tasks. The pretraining tasks are randomly sampled from a set of combinatorially many possible tasks using ShapeNet [2] objects. See Figure 8 in Appendix C.2 for examples. We expect this important clarification to change your evaluation of the contribution as well address your questions. Please inform us if we have misinterpreted your concerns or if you have further questions.