Paper: Motif: Intrinsic Motivation from Artificial Intelligence Feedback (ICLR-2024)

Reviewer comment (Novelty):
I think the point of this paper is that "joint optimization of preference-based and extrinsic reward helps resolve the sparse reward problems". As the source of feedback, either humans or LLMs are OK. I think describing this as LLM's contribution might be an overstatement.

Author response:
> I think the point of this paper is that "joint optimization of preference-based and extrinsic reward helps resolve the sparse reward problems". As the source of feedback, either humans or LLMs are OK. I think describing this as LLM's contribution might be an overstatement. [...] As a preference-based RL method, I guess there are no differences from the original paper [1].

We never claim that an LLM's feedback is inherently better than the feedback coming from humans, even though we believe assessing whether that could be the case is an interesting avenue for future work. Instead, we simply leverage an LLM's feedback because of its scalability: in just a few hours on a small-sized cluster, one can annotate thousands of pairs of games events, which would require significant amounts of human NetHack experts's labour otherwise. This scalability, leveraged also in recent work on chat agents (e.g., Constitutional AI from Bai et al, 2022), allows for a method like Motif to fully leverage human common sense to bootstrap an agent's knowledge.

Reviewer comment (Novelty):
As a preference-based RL method, I guess there are no differences from the original paper [1]. In the LLM literature, [2] leverages GPT-4 to solve game environments, and [3] incorporates LLM-based rewards for RL pretraining.

Author response:
> In the LLM literature, [2] leverages GPT-4 to solve game environments, and [3] incorporates LLM-based rewards for RL pretraining.

In [2] the Voyager algorithm uses complex prompt engineering involving significant amounts of human knowledge and engineering (such as deliberately prompting the LLM to use the “explore” command regularly). Additionally, Voyager relies on a Javascript API to bridge the gap between the LLM's language outputs and the high dimensional observations and actions of Minecraft. Finally, Voyager relies on perfect state information about certain features of the game (e.g. agent position and neighbouring objects). Voyager also builds on GPT-4's strong coding abilities, which would likely not be the case of current open models. Altogether, these factors strongly limit Voyager's general applicability and reproducibility. On the other hand, Motif relies on very limited human knowledge, being able to get significant performance even without any information about NetHack. Moreover Motif is way simpler to implement, with very few, clearly separated, moving pieces, providing a robust solution for leveraging prior knowledge from large models. This makes our approach a significantly more general method that has the potential to be applied to multiple domains or be possibly combined with powerful Large Vision Language Models.

We have additionally compared Motif to the ELLM approach of [3], adapted as described in the general response, showing that Motif significantly outperforms it in all NLE tasks. Please see the common response for a detailed description of the experiment. The paper also highlights in Section 5 (Related Work) the important differences between Motif and the ELLM algorithm. In particular, ELLM's reward function cannot, by design, exhibit the exploration and credit assignment properties of the one produced by Motif (see Section 4.2 of the paper, "Alignment with human intuition"). We believe those differences are key to the strong performance of Motif.

Reviewer comment (Writing):
Terminology: I'm not sure if a preference-based reward should be treated as an "intrinsic" reward. I think it is extrinsic knowledge (from humans or LLM).

Author response:
> Terminology: I'm not sure if a preference-based reward should be treated as an "intrinsic" reward. I think it is extrinsic knowledge (from humans or LLM).

As standard, we refer to extrinsic reward as the reward that comes from the task to be performed in the environment, whereas intrinsic rewards are provided by the algorithm. From that point of view, the reward provided by the LLM is intrinsic (as opposed to extrinsic) to the agent. Please notice that this terminology has previously been used in the literature (see, for instance, the ELLM [3] paper).

Reviewer comment (Reproducibility):
Which RL algorithm is used for Motif? I may miss the description in the main text.

Author response:
> Which RL algorithm is used for Motif? I may miss the description in the main text.


We use the asynchronous PPO implementation of Sample Factory (Petrenko et al., 2020). This information is available in the paper on the bottom of page 4. We chose this implementation as it extremely fast: we can train an agent on 2B steps of interactions in only 24h using only one V100 GPU.

Reviewer comment (Writing):
In Section 2, `... O the observation space ...` might be `... O is the observation space ...`.

Author response:
> (Minor Issue)

We thank the reviewer for spotting the typo. We corrected it in the updated version of the paper.

Reviewer comment (Experiments):
Can Motif be applied to other environments beyond the NetHack Learning Environment (NLE)?

Author response:
Thank you for your feedback!

>  Can Motif be applied to other environments beyond the NetHack Learning Environment (NLE)?

We could not investigate this question in our current paper, as it is already compact with detailed analysis on the behavior, the risks of using LLMs and the possibilities for defining diverse rewards. By adding other environments, we could not provide such in-depth analysis. 

We strongly believe that Motif is a general method, and it can applied to other environments after reasonably-sized efforts, when its main assumptions are satisfied. In particular, Motif's LLM needs to have enough knowledge about the environment, which is related to the presence on the Internet of text related to it, and the availability of an event captioning system. These assumptions are realistic in many environments, both when dealing with a physical system (e.g., a robot accompanied by a vision captioner) and a simulated/virtual world (e.g., a commonly-played videogames or Web browsing). Additionally, one could apply the general architecture of Motif to any environments based on visual observations, by just substituting a VLM in place of the LLM. We believe this is an exciting direction for future work.

To give some context on our choice of environment, NetHack is a challenging and illustrative domain to deploy an algorithm like Motif. Captions in NetHack are non-descriptive: they do not provide a complete picture on the underlying state of the game. Moreover, these captions are sparse, appearing in only 20% of states. This means that overall there is a high degree of partial observability. Despite this challenge Motif is able to thrive and show results that we have not witnessed in the literature previously. 

We believe that if we were to apply Motif in other environments with more complete descriptions we could see even stronger performance. This would bring important questions to be studied: what exactly is the impact of partial observability on preferences obtained from an LLM? Do more detailed captions unlock increasingly more refined behaviors from the RL agent? Such important questions could be investigated by future work.

Reviewer comment (Reproducibility):
What a RL algorithm is used for RL fine-tuning?

Author response:
> What a RL algorithm is used for RL fine-tuning?

We use the asynchronous PPO implementation of Sample Factory (Petrenko et al., 2020). This information is available in the paper on the bottom of page 4. We chose this implementation as it extremely fast: we can train an agent on 2B steps of interactions in only 24h using only one V100 GPU.

Reviewer comment (Experiments):
The paper could benefit from a more extensive comparison to other methods, especially those that also attempt to integrate LLMs into decision-making agents.

Author response:
First of all, we refer the reviewer Figure 7 in the Appendix F, in which we show that Motif outperforms four competitive baselines, including E3B (Henaff et al, 2022) and NovelD (Zhang et al., 2021), two state-of-the-art approaches specifically created for procedurally-generated domains such as NetHack. 

In the updated paper, we have now additionally added a comparison to ELLM (Du et al., 2023), a recent approach for deriving reward functions from LLMs, showing that Motif's performance significantly surpasses such LLM-based baselines across all tasks. This is due to the peculiar features of Motif's intrinsic reward (e.g., its anticipatory nature), which, by design, are not implied by a reward function based on the cosine similarity between a goal and a caption. Our implementation is described in detail in the general answer above.

Reviewer comment (Evaluation):
There is a lack of discussion on the computational cost and efficiency aspects of implementing Motif.

Author response:
Due to our implementation being modular and asynchronous (that will be publicly released), dataset annotation is not especially expensive. Please see the general response for complete computational considerations. In addition, in Figure 15 we show that the performance of our method is particularly robust to the size of the dataset of annotations: Motif is able to outperform the baselines even with a dataset that is five times smaller (i.e., ~100k annotations).

Reviewer comment (Evaluation):
While the paper makes a strong case for Motif, it doesn't delve deeply into the limitations or potential drawbacks of relying on LLMs for intrinsic reward generation.

Author response:
We believe, as the reviewer does, that addressing limitations in LLM-based work is critical: this is why a substantial fraction of our paper is devoted to analyzing limitations and pitfalls of intrinsic motivation from an LLM's feedback. In particular, we want to highlight that we dedicated a full page of the paper to demonstrating evidence for, explaining, and characterizing _misalignment by composition_, a negative phenomenon relevant to our framework, whose emergence is a current limitation of Motif. In addition, we studied the sensitivity of Motif to different prompts, showing in Figure 6c that semantically-equivalent prompts can lead, in complex tasks, to drastically different behaviors.  We believe this is a limitation of current approach based on an LLM's feedback, and hope that future work will be able to address it. Finally, we also included in Appendix H.3 a study on the impact of the data diversity of the dataset (through which we elicit preferences) and the resulting final performance.

Please notice that we are also very upfront about the fundamental assumption behind Motif, which is also a fundamental assumption behind the zero-shot application of LLMs to new tasks: that the LLM contains prior knowledge about the environment of interest. Our Introduction is centerered around this assumption.

Reviewer comment (Evaluation):
Could the authors offer insights into why agents trained on extrinsic only perform worse than those trained on intrinsic only rewards?

Author response:
Our paper already provides some insights in the "Alignment with human intuition" paragraph of Section 4.2, but we will now provide an additional perspective that can be beneficial to understand this result. By inspecting the messages preferred by the intrinsic reward function, one can quickly realize that the agent will receive from the LLM's feedback three kinds of rewards: _direct rewards_, _anticipatory rewards_ and _exploration-directed rewards_. Direct rewards (e.g., for "You kill the cave spider!") leverage the LLM's knowledge of NetHack, implying a reward very similar to the score (i.e., the extrinsic reward). Motif's reward, however, goes beyond this. Anticipatory rewards (e.g., for "You hit the the cave spider!") implicitly transport credit from the future to the past, encouraging events not rewarded by the extrinsic signal and easing credit assignment. Finally, exploration-directed rewards (e.g., for "You find a hidden door.") directly encourage explorative actions that will lead the agent to discover information in the environment. Together, these three types of rewards allow the agent to maximize a proxy for the game score that is way easier to optimize compared to the actual game score, explaining the increased performance.

Reviewer comment (Experiments):
What's the best strategy to optimally balance intrinsic and extrinsic rewards during training?

Author response:
We show in Figure 10c in Appendix that Motif is quite robust to how the two rewards are balanced. Broadly speaking, given that the nature of Motif's intrinsic reward brings it closer to a value function, future work can explore potentially more effective ways to leverage such type of intrinsic reward, for instance via potential-based reward shaping (Ng et al., 1999).

Reviewer comment (Evaluation):
How robust are agents trained with Motif against different types of adversarial attacks or when deployed in varied environments?

Author response:
Our experiments on prompt sensitivity (Figure 6c, 11b, 11c) can be interpreted as being close to this kind of study, showing the seemingly small variations of a prompt can trigger large or small variations of performance and behavior, depending on the environment. Future work should explore the possibility of studying the effect of actual adversarial attacks on prompts.

Reviewer comment (Experiments):
Given that one of the key contributions of the paper is the intrinsic reward function that is learnt from preferences extracted from the LLM, it might be worthwhile having a baseline that gives preferences using a simpler model (say sentiment analysis) and learn the RL policy using this intrinsic reward model.

Author response:
> it might be worthwhile having a baseline that gives preferences using a simpler model (say sentiment analysis) and learn the RL policy using this intrinsic reward model.

We added the results of an experiment using a sentiment analysis model as a preference model in the updated paper (Figure 8 of Appendix A.6). We use a [T5 model](https://huggingface.co/mrm8488/t5-base-finetuned-imdb-sentiment) fine-tuned for sentiment analysis, and extract, for each message, a score computed as the sigmoid of the confidence of the model in its positive or negative sentiment prediction. Then, for each pair in the dataset, we assign a preference based on the message with higher sentiment score. Finally, we execute reward training and RL training as with Motif. 

Results on the `score` task show performance close to zero, both with and without extrinsic reward. This poor performance can be easily explained: a generic sentiment analysis model cannot capture the positivity or negativity of NetHack-specific captions. For instance, killing or hitting are generally regarded as negative statements, but they become positive in the context of killing or hitting monsters in NetHack. Llama 2 can understand this out-of-the-box without any fine-tuning, as demonstrated by our experiments. Also note that such a vanilla sentiment analysis model cannot be easily steered, thus losing any opportunity for the controllability offered by Motif.

To attest to Motif's strong performance, we also compared with an additional LLM-based baseline (as requested by Reviewer VWkY). The details of this additional experiment are presented in the common response above.

Reviewer comment (Experiments):
In the part on "Alignment with human intuition", the paper mentions that the agent exhibits a natural tendency to explore the environment by preferring messages that would also be intuitively preferred by humans. Is this a consequence of having a strong LLM, or is it due to the wording of the prompt?

Author response:
> the paper mentions that the agent exhibits a natural tendency to explore the environment by preferring messages that would also be intuitively preferred by humans. Is this a consequence of having a strong LLM, or is it due to the wording of the prompt?

We believe this is due to the fact that the LLM was pretrained on massive amounts of human data, and then fine-tuned on human preferences. Indeed, even when using the zero-knowledge prompt presented in Prompt 2 of the Appendix B, Motif's reward function allows agents to play the game effectively even without any reward signal from the environment (see Figure 6b).

Reviewer comment (Experiments):
An ablation over $\alpha_2$ has been provided in the appendix, but the value of the coefficient for the intrinsic reward $\alpha_1$ is kept fixed at 0.1; could you explain the reason behind that?

Author response:
> An ablation over $\alpha_2$ been provided in the appendix, but the value of $\alpha_1$ the coefficient for the intrinsic reward is kept fixed at 0.1; could you explain the reason behind that?

The important factor when combining two terms in a reward function is the relative weight given to each one of them.  We decided to ablate by varying $\alpha_2$ to progressively give more weight to the extrinsic reward, compared to the intrinsic reward. This allows us to show that, while Motif already performs well in the absence of extrinsic reward (i.e., for $\alpha_2=0$), adding progressively more importance to the reward signal coming from the environment (by increasing $\alpha_2$) correspondingly increases performance, but only up to a point. In the moment at which the relative weight given to the intrinsic reward becomes too small, the performance starts degrading as the agent acts essentially more and more as the extrinsic-only baseline.

Reviewer comment (Evaluation):
In Figure 6c, the score for the reworded prompt is quite low but its dungeon level keeps steadily rising compared to the default prompt. Is this a case of the agent hallucinating its dungeon level due to a very high intrinsic reward, or is it something else?

Author response:
> In Figure 6c, the score for the reworded prompt is quite low but its dungeon level keeps steadily rising compared to the default prompt. 

Figure 6c shows that a rewording of the prompt in a task as complex as the one of finding the oracle can cause a complete change in the strategy followed by the agent. In the case of the original prompt, the agent hacks the reward, solving the task without the need of going down the dungeon. When using the reworded prompt, the agent instead starts going down the dungeon to look for the oracle, and finds it for up to 7% of the episodes. The plot thus shows the sensitivity of systems like Motif to variations of the prompts that could be perceived as small to humans. In the paper we put a strong emphasis on understanding such changes in behavior as we believe it to be fundamental if we are to release any LLM-based agent in more realistic situations. As a side note, we corrected the y axis label and normalization in Figure 6c in the updated paper to be consistent with the rest of the paper (from "Score" to "Success Rate").