Paper: Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
**Real world impact**  Although the work shows the potential of VLM-RMs, all conducted experiments have been based on relatively simple and synthetic tasks, rather than from the real world. To further increase the potential impact of this work, it would be beneficial to consider some more tasks with real-world applications.

Author response:
> Real world impact: Although the work shows the potential of VLM-RMs, all conducted experiments have been based on relatively simple and synthetic tasks, rather than from the real world.

While it is true that we focused on synthetic tasks in a standard control environment, we do find evidence that VLM-RMs become significantly better the more visually realistic the environments are. Based on this, we would predict our method works even better in more practical environments. Together with the strong positive scaling with model size that we found, we think this suggests our method is highly scalable. Based on these results, we believe we demonstrate a very high potential for real-world impact, certainly compared to typical ML papers.

Of course it is important to validate this hypothesis in practical environments, and we are planning to address this in future work.

Reviewer comment (Experiments):
When using the VLMs as the reward function for a task, one may need to consider if one should use it as a dense reward signal or a terminal reward, can the authors elaborate on how they considered this question?

Author response:
> When using the VLMs as the reward function for a task, one may need to consider if one should use it as a dense reward signal or a terminal reward, can the authors elaborate on how they considered this question?

This is a good question and it is indeed not always clear how to make this choice.

The main reason we chose to provide dense rewards is that we found empirically (e.g., in the classic control environments) that the VLM rewards are well shaped, i.e., measure progress towards the goal. If the rewards already have this property, we might as well provide them as dense rewards to make exploration easier.

Another reason to provide dense rewards is that you might want to specify constraints that have to hold during the entire trajectory. For example we could specify that a robot should pick up an object and keep distance from a wall. Then we want to make sure to evaluate every state according to if it reaches the goal but also how close you are to the wall. And we certainly want to provide the latter as a dense reward.

We hope, this clarifies any open questions about our work!

Reviewer comment (Novelty):
The proposed approach of using a CLIP model 0-shot to specify language-conditioned rewards is not novel, and the relevant references are cited in the related work section (Cui et al., Mahmoudieh et al.). The main technical novelty is in proposing the baseline regularization. However, the main experiment on learning humanoid posing tasks does not use this regularization at all. As such, it appears that the used approach in these experiments is identical to those proposed in prior work (Mahmoudieh et al.), just applied on a different task.

Author response:
Thanks for this detailed review! Your comments will certainly help to improve our paper!

Before answering specific questions, let us make a general point on novelty. While our paper does not propose an entirely new method, it is the first paper to make this method work for learning complex tasks. This positive result which stands in contrast to previous weaker negative results is an important contribution and might make a difference in whether people actually use this method or not. The goal of empirical research in ML should be to produce evidence on which methods work and not only to develop new methods.

> How does the evaluated approach for Table 1 differ from prior works like Mahmoudieh et al. that also use 0-shot CLIP embeddings for language-conditioned reward computation?

The primary method proposed by Mahmoudieh et al. uses task-specific finetuning. This makes it much more expensive and less general than the zero-shot method we focus on. They do briefly consider a method they call “Vanilla Dot-product Visuolinguistic Base Reward Model” as a baseline which is similar to ours.

Importantly Mahmoudieh et al. conclude this baseline does not work well, motivating the design of their more task-specific method. For example, in Section 2.1. they say: “ One significant limitation of CLIP is that it cannot distinguish spatial relationship of objects in images. This limits our base reward model from being useful for tasks that have spatial goals. Our full zero-shot reward model remedies this issue by leveraging CLIP in a very different way.”

Unfortunately, Mahmoudieh et al. do not provide code to reproduce the experiments. But from the paper, three key differences stick out:
- **Reward formulation**: Their base reward model uses an (unnormalized) dot product as a reward, whereas we use a cosine similarity (i.e., a normalized dot product). Cosine similarity is more aligned with the CLIP training objective, so it should work better.
- **Model size**: They use the RN50 CLIP model, which is the smallest model out of the ones we use. One of our key findings is that model size is very important for CLIP models being useful zero-shot reward models.
- **Training**: They train for 200k steps, whereas we train for 10M steps. We also use SAC over PPO, and the former is typically more sample efficient.

Reviewer comment (Novelty):
The evaluated approach for Table 1 is not clearly differentiated from prior works like Mahmoudieh et al., which also use 0-shot CLIP embeddings for language-conditioned reward computation.

Author response:
> How does the evaluated approach for Table 1 differ from prior works like Mahmoudieh et al. that also use 0-shot CLIP embeddings for language-conditioned reward computation?

The primary method proposed by Mahmoudieh et al. uses task-specific finetuning. This makes it much more expensive and less general than the zero-shot method we focus on. They do briefly consider a method they call “Vanilla Dot-product Visuolinguistic Base Reward Model” as a baseline which is similar to ours.

Importantly Mahmoudieh et al. conclude this baseline does not work well, motivating the design of their more task-specific method. For example, in Section 2.1. they say: “ One significant limitation of CLIP is that it cannot distinguish spatial relationship of objects in images. This limits our base reward model from being useful for tasks that have spatial goals. Our full zero-shot reward model remedies this issue by leveraging CLIP in a very different way.”

Unfortunately, Mahmoudieh et al. do not provide code to reproduce the experiments. But from the paper, three key differences stick out:
- **Reward formulation**: Their base reward model uses an (unnormalized) dot product as a reward, whereas we use a cosine similarity (i.e., a normalized dot product). Cosine similarity is more aligned with the CLIP training objective, so it should work better.
- **Model size**: They use the RN50 CLIP model, which is the smallest model out of the ones we use. One of our key findings is that model size is very important for CLIP models being useful zero-shot reward models.
- **Training**: They train for 200k steps, whereas we train for 10M steps. We also use SAC over PPO, and the former is typically more sample efficient.

Reviewer comment (Experiments):
In Mahmoudieh et al., the non-finetuned model does not work at all as a reward function — it is unclear whether the fact that the model in the submission works well is purely based on the chosen task domain.

Author response:
> In Mahmoudieh et al. the non-finetuned model does not work at all as a reward function -- is the fact that the model in the submission works well purely based on the chosen task domain?

We suspect that all of the above mentioned differences are important for our method to work well. Specifically, we show in Figure 4 that model size is critically important, and we observed that for less than 1M training steps we see barely any training progress. While we did not test the (unnormalized) dot-product formulation systematically, early experiments suggested that it might be more noisy. Certainly, it is conceptually less principled because CLIP is trained with a cosine similarity objective.

We would have liked to evaluate our method in the environments studied by Mahmoudieh et al. Unfortunately, they use non-standard tasks and do not provide code, so it would be difficult to reproduce their setup without significant effort. We agree that it is an open question how well VLM-RMs will work in robotics domains where many tasks are of a more “spatial” nature, and we are actively interested in studying this in future work.

Reviewer comment (Experiments):
Based on the rebuttal, the paper claims three main “tricks” to make things work: (1) model size — this one makes sense and is validated with experiments; (2) reward formulation — this one is not validated experimentally; (3) choice of RL optimizer algorithm — this one seems orthogonal to the core question of how to formulate reward functions and is also not experimentally validated.

Author response:
Unfortunately, Mahmoudieh et al. do not provide code to reproduce the experiments. But from the paper, three key differences stick out:
- **Reward formulation**: Their base reward model uses an (unnormalized) dot product as a reward, whereas we use a cosine similarity (i.e., a normalized dot product). Cosine similarity is more aligned with the CLIP training objective, so it should work better.
- **Model size**: They use the RN50 CLIP model, which is the smallest model out of the ones we use. One of our key findings is that model size is very important for CLIP models being useful zero-shot reward models.
- **Training**: They train for 200k steps, whereas we train for 10M steps. We also use SAC over PPO, and the former is typically more sample efficient.

We suspect that all of the above mentioned differences are important for our method to work well. Specifically, we show in Figure 4 that model size is critically important, and we observed that for less than 1M training steps we see barely any training progress. While we did not test the (unnormalized) dot-product formulation systematically, early experiments suggested that it might be more noisy. Certainly, it is conceptually less principled because CLIP is trained with a cosine similarity objective.

Reviewer comment (Novelty):
Simply proposing one regularization is not novelty enough unless the authors can prove it can boost performance in a wide range of tasks.

Author response:
> As for contribution, I admit the proposed method is somewhat interesting, however, I still do not believe it reaches the bar of ICLR. This paper wants to remove the irrelevant parts from the environment if I understand correctly.

We want to further clarify that “goal-baseline regularization”, while the main methodological novelty, is only part of our paper’s contribution.

Fundamentally, our paper is an empirical study of Vision-Language Models as Reward Models _in a zero-shot manner_. Our results include novel and impactful insights _even for the most basic VLM-RM approach_ without goal-baseline regularization. We show in our work that, without expensive fine-tuning or using other complex techniques, Vision-Language Model rewards can be **well-shaped, aligned with human labels, and scale predictably** in quality with size.

Previous work, such as Mahmoudieh et al., has only tried similar methods with small CLIP methods, and found them to not work robustly. In contrast, we show that to obtain useful rewards, the most important variable is model size. We find it extremely important to report these findings to demonstrate that VLM reward models can work well at larger scales previously not investigated.

Most importantly, **we are the first to show how to learn complex tasks with VLM-RMs that do not have a ground truth reward function**!

Goal-baseline regularization is a novel, simple method for obtaining higher performance from VLM-RMs in a zero-shot manner without adding too much complexity. While it’s an important contribution of the paper, we think it has to be considered in the context of the points above.

We hope that this clarifies the scope of the paper’s contributions, potentially leading the reviewer to reconsider their score.

Reviewer comment (Experiments):
The experiments are not sufficient. Taking Figure 3 as an example, more seeds and more tasks should be included for baselines.

Author response:
> The experiments are not sufficient. Taking Figure 3 as an example, more seeds and more tasks should be included for baselines.

We focus on learning tasks that do not have a ground truth reward function, so we need human evaluation. While we agree that more seeds and tasks would be desirable, evaluation tends to be quite expensive in our setting.

Specifically, note that Figure 3 is an ablation of some features of our setup but still requires human evaluation. Running this ablation for many tasks and seeds would be very expensive and likely provide limited new insights.

We think we did quite extensive experiments in simple and complex environments. For example, we thoroughly studied effects of different model sizes and goal-baseline regularization and performed different ablations. **Could you please clarify specifically which parts of our experiments you find lacking?**

We hope the response addresses some of your concerns, and we would appreciate any clarification of your specific concerns, to allow us to respond to them.

Reviewer comment (Experiments):
Risk of bias in human experiments. While this is duly acknowledged in the paper, the fact that the human analysis was conducted by one of the authors of the paper is a potentially strong source of bias in the human evaluation.

Author response:
This is true and a valid criticism. To reduce bias in the evaluation, we will publish the full raw videos we evaluated to produce the numerical results together with the paper (rather than only the samples currently provided on the website).

Reviewer comment (Novelty):
Suggested citation if authors believe it is relevant (using VLMs to improve exploration in RL): https://openreview.net/forum?id=-NOQJw5z_KY

Author response:
Thanks for the suggestions. We will add it as useful evidence on VLM representations providing useful information for RL algorithms.

Reviewer comment (Experiments):
The paper states that the "CLIP rewards are only meaningful and well-shaped for environments that are photorealistic enough for the CLIP visual encoder to interpret correctly," yet the paper focuses on control environments without realistic visuals. Why not instead focus on more visually realistic simulation benchmarks and not have to modify the environment to make the rendering more realistic to fit the algorithm?

Author response:
We found the MuJoCo humanoid environment to be a good starting point because it is a standard control benchmark and relatively inexpensive to set up, run and render. This environment provides a fast feedback loop to test different aspects of the algorithm (like scale and baseline prompts) but it is still complex enough to allow for interesting tasks. We discovered only through our experiments that realism of the visual representation is quite important. Based on our results, we would predict that VLM-RMs work even better in visually more realistic environments and we’re planning to test this in follow-up work.

Reviewer comment (Experiments):
It's unclear if the goal-baseline regularization is necessary. The primary experiments in Table 1 don't use any goal-baseline regularization. The analysis in Fig. 4a shows the maximum or minimum amount of goal regularization to be best. Additionally, to what degree can the same effects of the goal-baseline regularization be incorporated in the goal prompt? A more detailed goal prompt can just specify to ignore the irrelevant information.

Author response:
Recall that lower EPIC distance indicates a better correlation with the human labels, i.e., a better reward model. Fig. 4a suggests that typically a regularization strength between 0 and 1 is better than the extremes 0 or 1. Only for RN50 (the smallest CLIP model we tried), the maximum amount of regularization seems to be best. Importantly, in none of the cases the minimum amount of regularization seems to be best.

However, the effect of the regularization is weaker for larger models, and the difference in our results is not always significant. We interpret this to mean that goal regularization can help a lot with reward shaping when using smaller models, but might be unnecessary when using large enough models.

This is a great question! In principle, we can encode all necessary information in the prompt. In practice, this is difficult with current models. For example, it’s well documented that current CLIP models can struggle with negations which would be necessary to describe the contrast between the goal and the baseline prompt. Hence, it is better to use the embeddings of a goal and baseline prompt and encode the negation as a subtraction as in our goal-baseline regularization. But we do expect future VLMs to get better at handling negations and other complex prompts.

Reviewer comment (Experiments):
While the paper shows the strong final results of VLM-RM, the RL training stability of VLM-RM is unclear. Is it easy for RL methods to learn from the VLM-RM reward? How does this compare to a ground truth reward? See my further comments under the questions section.

Author response:
We can plot learning curves of the CLIP rewards, but, as you say, they don’t necessarily show actual performance. While we can evaluate the ground truth reward in the classical control tasks, this is not possible for the humanoid tasks where we specifically chose “semantic” tasks with no programmatic ground truth function.

EPIC distance is only a measure of distance between human reward labels and the CLIP reward labels, but it also does not allow us to show the policies’ learning progress during training.

That being said, we agree it would be good to provide some learning curves in the paper to evaluate training stability, and we will do so (we will add them to the appendix). In brief: we find some variation depending on the initialization of the policy (for some random seeds the policies learn poorly), but overall training is relatively stable and comparable to the ground truth reward for task where we have one.

For now, we have uploaded a PDF containing training curves for all experiments at the following link: https://drive.google.com/file/d/1W1ERoHZugbpi0Ap2yC45qjtf8PryT11x/view

As you can see the training is reasonably stable, and in cases where we have ground truth reward functions, the CLIP training curve closely mirrors the ground truth training curve. When revising the paper, we will add these plots and make sure to point to the training curves when discussing training stability.

Reviewer comment (Reproducibility):
What exact values are selected for $\alpha$ in Fig. 4a? I recommend putting an indicator in this plot to show which values are used.

Author response:
Good point, we’ll add indicators to the figure. The values for alpha are: 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.99.

Reviewer comment (Writing):
What is the optional context $c$ in Eq. 1 used for? I don't see it referred to later in the paper.

Author response:
In general, we define the context as a “catch-all” term containing information provided to the VLM that is not the goal prompt. In our case this only happens via the baseline regularization technique. So, you can think of the context in our implementation as containing the baseline prompt and regularization strength. We’ll clarify this in the paper. In other kinds of VLM-RMs involving vision-augmented language models (not yet tested by us), the context could include a system prompt, used to make the model to act as a reward model.

Reviewer comment (Evaluation):
Can the paper include the RL learning curves for all the experiments showing the number of samples versus either the predicted CLIP return or a ground truth metric, like true reward in the classic control tasks or EPIC in the humanoid tasks? Given there is a ground truth reward in the classic control tasks, can the authors also include that as a reference for the VLM-RM learning curves? It is important to see the training stability and efficiency under the CLIP model reward.

Author response:
We can plot learning curves of the CLIP rewards, but, as you say, they don’t necessarily show actual performance. While we can evaluate the ground truth reward in the classical control tasks, this is not possible for the humanoid tasks where we specifically chose “semantic” tasks with no programmatic ground truth function.

EPIC distance is only a measure of distance between human reward labels and the CLIP reward labels, but it also does not allow us to show the policies’ learning progress during training.

That being said, we agree it would be good to provide some learning curves in the paper to evaluate training stability, and we will do so (we will add them to the appendix). In brief: we find some variation depending on the initialization of the policy (for some random seeds the policies learn poorly), but overall training is relatively stable and comparable to the ground truth reward for task where we have one.

For now, we have uploaded a PDF containing training curves for all experiments at the following link: https://drive.google.com/file/d/1W1ERoHZugbpi0Ap2yC45qjtf8PryT11x/view

As you can see the training is reasonably stable, and in cases where we have ground truth reward functions, the CLIP training curve closely mirrors the ground truth training curve. When revising the paper, we will add these plots and make sure to point to the training curves when discussing training stability.