Paper: OMNI: Open-endedness via Models of human Notions of Interestingness (ICLR-2024)

Reviewer comment (Theory):
What if the LLM does not have the human prior or reasoning capacity about the task "interestingness" on the given domain?

Author response:
Given the extensive dataset that LLMs have been trained on, it is unlikely that LLM would not already have some prior on any task that is human-generated. Regardless, we believe OMNI being helpful on many/ most tasks is still valuable, even if it is not helpful on all. 

In the case where the LLM does not have a good enough notion of interestingness, we propose to use human feedback to train (or fine-tune) a model of interestingness. This is termed Open-Endedness with Human Feedback (OEHF) in Section 6 (Section 5 of the original manuscript), as part of future work. Similar to Reinforcement Learning with Human Feedback (RLHF), one could potentially train a model that can effectively capture an ineffable property that, although challenging to quantitatively measure, is readily identifiable upon observation (e.g., a backflip for RLHF, or whether a task is interesting for OEHF).

Reviewer comment (Evaluation):
Based on my personal experiences of training agents in Crafter, an agent with a random task generator (Uniform) should not be that bad as it should be able to master not just a limited set of tasks as the curve in Figure 3 suggested. Can the authors clarify this?

Author response:
If all of the tasks are interesting (i.e., only the 15 tasks designed in the original Crafter environment), using uniform sampling would be sufficient. This effectiveness can be attributed to the fact that these environments are often already designed with some human intuition for interestingness, only including tasks or features that we instinctively find interesting.

However, in the broader context of open-ended learning where we want environments with infinite possible tasks, it is extremely difficult to design an environment or task space that only contains interesting features. Take a kitchen environment for example, there can be endless variations of possible tasks (e.g., putting a cup in slightly different positions), and it is almost impossible to curate a set of tasks that are all interesting. We modified Crafter and BabyAI environments by expanding the task space beyond the carefully selected few in order to emulate this phenomenon from vast, open-ended environments. It is because there are many boring or impossible tasks in the modified versions of these domains that uniform sampling fails.

In new experiments, we demonstrate OMNI’s ability to handle just such an infinite task space (the set of all tasks specifiable in natural language). Please see our general reviewer response for a summary.

Reviewer comment (Experiments):
If my understanding is correct, LP seems to be more crucial to the success of mastering more challenging tasks as it takes the learning process into consideration, while "interestingness" is about not wasting time on tasks that could not lead to meaningful learning. Therefore, if an unlimited round of learning is allowed, an agent with LP only should ultimately match the results of full OMNI on the average success rate. However, this is not the case in all the experiments. Maybe the LP baseline is not run for enough rounds? I would like to learn more about your thoughts on this.

Author response:
In an ideal scenario with infinite compute, an agent guided solely by LP could converge to performance levels of an agent utilizing the full OMNI approach. However, in reality, having infinite compute is impossible. In a truly open-ended setting, there are an infinite number of uninteresting tasks that LP will inevitably get distracted by. For example in the real world, there can be endless variations of mathematical, or even addition, questions (e.g. just by changing constants) that the agent could attempt. In the Crafter environment, LP is distracted by boring repetitive tasks, which require substantial compute and training steps, whereas OMNI is able to filter out the boring tasks and only focus on the interesting tasks to gain more meaningful learning.

Reviewer comment (Experiments):
The baselines compared in this paper are not strong enough. All baselines are intuitively set, and some previously proposed methods (some are discussed in related work, like curriculum learning for RL) need to be included.

Author response:
Learning progress (LP) is an RL curriculum learning method, which has been demonstrated by previous works as a strong baseline (Kanitscheider et al., 2021). Using LP as an RL curriculum, Kanitscheider et al. (2021) could train Minecraft agents to collect diamonds from scratch, which has been recognized as an extremely challenging RL task. We will make this clearer in the manuscript.

We have added more experiments to our study. These new experiments, like related works (e.g., intrinsic motivation, novelty search, diversity), rely on pre-specified definitions of what counts as interestingly different. We compare OMNI with an alternative MoI which relies on predefined heuristics based on sentence embeddings. In this alternative, embedding-based MoI (eMoI), interestingness is determined by the distances between sentence embeddings of task descriptions. We show where such predefined metrics fail, and that OMNI outperforms the controls significantly. Please see Appendix P in the revised manuscript for the full details and results.

Reviewer comment (Theory):
It might not be practical to know all candidate tasks in advance, and just let the large model choose one. In the RL setup, the agent needs to explore the environment and find out all candidate tasks.

Author response:
Thank you for pointing this out. We also knew that such a domain was the next frontier for OMNI (we planned on it being a follow-up paper, but are happy to add it to this paper instead). So immediately after submitting the original version, we began testing OMNI in an infinite, complex, robotics domain, and are pleased to report that OMNI far outperforms controls (the set of all tasks expressible in natural language). The revised manuscript includes these results, please see our general reviewer response for a summary.

Reviewer comment (Evaluation):
Human alignment also needs to be discussed. This paper just uses the interestingness of large models as that of humans to do exploration. On the other hand, alignment is an unsolved question that targets aligning humans and large models and avoiding dangerous actions. Methods like RLHF are proposed here. Considering that human alignment is not perfect now, how can the interestingness of large models be used for choosing tasks while also ensuring important aspects like safety?

Author response:
We agree that human alignment and safety are important and interesting research challenges for open-endedness. We have added a discussion on this topic in Appendix R of the revised manuscript.

Reviewer comment (Novelty):
The missing related work (https://arxiv.org/pdf/2302.06692.pdf) uses language models to suggest plausibly useful goals, which shares a similar idea with this paper.

Author response:
Du et al. (2023) was discussed in Section 2.3 of Related Works in the original manuscript. Please let us know if there are more comments regarding it that you feel we should add.

Reviewer comment (Experiments):
Why is the survival component removed in the Crafter environment? This is quite important in a practical open-ended environment. Without considering the survival component, the aspects that need to be considered during choosing the task become less adequate and complex, which goes against the motivation of using LLMs here.

Author response:
By removing the survival component, task completion is less susceptible to external stochastic events like whether a skeleton has killed the player. It removes the need for the agent to learn and continually apply survival tactics against enemies or for food gathering. In other words, it allows us to focus on evaluating the tasks specified with less evaluation noise.

For completeness, based on your request, we conducted new experiments with the original Crafter environment, where survival components are retained. The findings from these experiment setup are consistent with our initial results, where OMNI outperforms Learning Progress and Uniform sampling significantly. For more details and results, please see Appendix O of the revised manuscript.

Reviewer comment (Reproducibility):
It’s better to show instructions used in determining interesting tasks. How to query the large model and what kind of results are obtained from it? An example here might be helpful.

Author response:
Full LLM prompts are provided in Appendices B and C. Appendix F (Appendix D in original manuscript) showcases a step-by-step example of how the model declares tasks as interesting or not. We have also added pseudocode in Section 3 of the revised manuscript for additional clarity.

Reviewer comment (Evaluation):
The performance of OMNI and other baselines on BabyAI is extremely bad (success rate < 0.008). Have the authors taken a look at the policies and understood what is happening? I understand this includes hard tasks, but have the authors considered plotting the results for only interesting tasks, similar to Crafter?

Author response:
As you guessed, the average success rate is low because it is an average across the entire taskset of 1364 tasks. Most tasks are very difficult, which are not even completed at a success rate of 0.05 at any time step by any method. The reason why the tasks are so difficult in BabyAI is that each task does not depend on the target object(s) or environment configuration in that episode. For example, the target object distribution for a GoTo task can be any object in the environment distribution. If the object is in another locked room, the GoTo task would implicitly require the agent to find a key, unlock and open the door, before navigating to the target object.

We like your suggestion of plotting on interesting tasks only. We did a version of that by averaging success rates over tasks with instruction lengths of one or two only (see additional Figure 15 in Appendix K). OMNI significantly outperforms controls, and shows a more impressive completion rate of 16%.

Reviewer comment (Experiments):
In the experiments, the authors use a pre-defined set of interesting, boring and challenging tasks. However, it would also be interesting to try the methods in a more open-ended setting whereby they use a random task generator that procedurally generates virtually unlimited number of tasks.

Author response:
Thank you for the suggestion. We are equally as excited to share the results of OMNI in an infinite, complex, robotics domain, and are pleased to report that OMNI far outperforms controls (the set of all tasks expressible in natural language). The revised manuscript includes these results, please see our general reviewer response for a summary.

Reviewer comment (Evaluation):
The evaluation of the trained models is performed only on the previously seen tasks. I was wondering if the authors considered evaluating on held-out, and particularly out-of-distribution (OOD) tasks to assess the robustness of the trained methods.

Author response:
In the current version, we are not claiming that OMNI can generalize to OOD tasks by training on interesting and learnable tasks. We agree that testing this would be an interesting future research direction, especially once OMNI is trained at scale.

Reviewer comment (Presentation):
A disproportionate emphasis on potential implications and future work in the paper might obscure the immediate contributions. For example, the Conclusion is > 1.25 page, mostly discussing future work. While this is very interesting, it feels to me that it takes the space from what actually was important in this paper and was moved to the Appendix. In the end, the reader has to go back and forth from main part of the paper to appendix to follow the ideas described in these different sections. It is my recommendation that the authors carefully consider what they want to include in the main part of the paper vs the appendix.

Author response:
We apologize for the need to switch between the main text and the appendix. If accepted to ICLR, we will commit to moving some of the experimental details back to the main text, and some future work to the appendix.

Reviewer comment (Novelty):
The statement "OMNI has the potential to significantly enhance the ability of AI to intelligently select which tasks to concentrate on next for endless learning and could pave the way for self-improving AI and AI-Generating Algorithms" reflects a giant gap between what the authors presented in this work compared to what they claim it has the potential of doing. I would recommend removing/rewording such statements.

Author response:
We have toned down some of the writing style and claims in the revised manuscript.

Reviewer comment (Experiments):
Unfortunately, the experimental section of the paper does not live up to the ambitious ideas described in the intro. Only two (fairly simple) environments are considered (Crafter and one MiniGrid task), and the experiments feel contrived. For Crafter, "Impossible" tasks are generated by simply assigning the agent 0 reward. "Boring" tasks are generated by asking the agent to repeat the same task several times. Because of this, it is possible to solve the Crafter setup with very simple heuristics, and an LLM is not really needed. Granted, LLMs are more general, but it would be a lot more compelling if this method were evaluated on tasks which are not solvable with simple heuristics.

Author response:
It is imperative to note that transitioning to such environments entails significant challenges. If we insist that the first paper of every new idea be showcased on a very complex domain, (1) we would really disadvantage academic labs, who do not have the compute resources often required for such complex domains, and (2) require many years of work before a student can publish their first paper.

Voyager (Wang et al., 2023) was done by a team of full-time scientists from NVIDIA, and probably has orders of magnitudes more compute than any academic lab. We purposefully chose Crafter, the academic equivalent of Minecraft, because it has similar dynamics but is within budget for an academic lab. Furthermore, both Crafter and MiniGrid have been shown to be challenging RL domains (Hafner, 2021; Du et al., 2023; Chevalier-Boisvert et al., 2018; Mu et al, 2022).

Nonetheless, there is value in extending OMNI to a more complex, open-ended domain. Hence, our revision includes additional results on a very complex, open-ended, realistic embodied kitchen robotics domain, AI2-Thor (Kolve et al., 2017). Please see our general reviewer response for a summary.

Reviewer comment (Reproducibility):
The paper's presentation needs improving. The paper is missing a lot of details about the algorithm. While Figure 1 gives a high-level overview of the system, it is still fairly vague and the rest of the description is all in the text and not very precise. It would be helpful to have the entire algorithm spelled out in pseudocode (in the main text), the exact process by which tasks are chosen described in equations or pseudocode, etc.

Author response:
We like your idea of including pseudocode for the entire algorithm, and have included it in Section 3 of the revised manuscript. Furthermore, as promised at the start of the Appendix, we will open-source the code to provide full transparency.

Reviewer comment (Experiments):
My main suggestion for improving this paper is to test on one or more challenging environments. While artificial settings like the ones used in this paper are nice for proof of concepts or ease of interpretation, they need to be complemented by experiments on a more challenging domain where simple heuristics are insufficient. I would suggest trying this either on Minecraft, which has been used for open-ended research [1], or NetHack, which is another very complex open-ended game where the extrinsic reward is insufficient (it is also fast to run) [2].

Author response:
It is imperative to note that transitioning to such environments entails significant challenges. If we insist that the first paper of every new idea be showcased on a very complex domain, (1) we would really disadvantage academic labs, who do not have the compute resources often required for such complex domains, and (2) require many years of work before a student can publish their first paper.

Voyager (Wang et al., 2023) was done by a team of full-time scientists from NVIDIA, and probably has orders of magnitudes more compute than any academic lab. We purposefully chose Crafter, the academic equivalent of Minecraft, because it has similar dynamics but is within budget for an academic lab. Furthermore, both Crafter and MiniGrid have been shown to be challenging RL domains (Hafner, 2021; Du et al., 2023; Chevalier-Boisvert et al., 2018; Mu et al, 2022).

Nonetheless, there is value in extending OMNI to a more complex, open-ended domain. Hence, our revision includes additional results on a very complex, open-ended, realistic embodied kitchen robotics domain, AI2-Thor (Kolve et al., 2017). Please see our general reviewer response for a summary.

Reviewer comment (Evaluation):
My second suggestion is improving the presentation (see my comments above). It's also not clear what criteria the agent uses to decide it can do a task "well" - is this based on success rate or reward? Please clarify. In addition to this, I think the Conclusion/Discussion/Future Work section is too long at over 1 page. This saved space can be used for extra experiments in this paper.

Author response:
Appendix F (Appendix D in original manuscript) explains how the agent decides which tasks are done well. On top of the details in the original manuscript, we hope that the additional pseudocode clarifies this. Reviewer LSLH indicated that the presentation was excellent and Reviewer CZLW commented that “the paper is clear and well-written”. Given the additional updates, we hope you find the presentation better. If accepted to ICLR’24, we will commit to moving some of the experimental details back to the main text, and move some future work to the appendix. If there are any other missing details, please let us know and we will add them to the manuscript.