Paper: Eureka: Human-Level Reward Design via Coding Large Language Models (ICLR-2024)

Reviewer comment (Theory):
Unrealistic assumption of access to the environment source codes: The reward code generation in this paper critically depends on having access to the source code of the MDP specification as context for the initial reward proposal. The authors have presented this as a benefit, allowing the LLM to exploit code structure to understand task environments. However, it makes an unrealistic assumption, as most reinforcement learning setups only require access to a black-box simulation. A significant limitation of this approach is that it may be infeasible for real-world robotic tasks where the transition function may either be unavailable or in different analytical forms. Given the heavy dependence on source code for task environment understanding, this method could be essentially restricted to simulated RL environments only.

Author response:
We would like to clarify that the problem setting of our work is not reinforcement learning, but rather reward design. This is traditionally done by humans who write and refine reward function codes in trial-and-error fashion in order to induce the desired behavior; Eureka automates this at scale. Its outer loop is a code generation problem, and it is standard to assume access to parts of the source code in order to generate/complete other parts of the code [1]. Furthermore, as we have clarified in our updated manuscript, Eureka’s “environment as context” procedure requires only the environment’s state and action variables to be exposed. This information can be supplied, for example, via an API without requiring the full environment code; see Example 1 in Appendix E, where we showed that a documentation-style description of the environment variables is sufficient for Eureka to generate effective reward functions. Eureka’s inner-loop does use RL to evaluate its reward candidates, and this inner-loop indeed only accesses the environment with Eureka generated reward function as black-box simulation. We hope that this explanation helps clarify the misunderstanding regarding Eureka’s use of environment as context and the problem setting it tackles.

Reviewer comment (Theory):
Strong assumption on the fitness function F(.): The evolutionary search for the LLM generated reward function requires a fitness function capable of assessing the quality of each proposed reward function. In this work, the fitness function F(.) is implicitly assumed to have access to the ground truth reward function to evaluate the induced policies of the proposed reward functions. This limitation implies that the method is applicable only to tasks that come with known ground-truth reward functions. This weakens the method's practical applicability, restricting it mainly to idealized or known environments, hindering its usefulness for real-world, less predictable reinforcement learning tasks.

Author response:
We would like to clarify that the ground-truth reward functions are simply sparse task success criteria, which are easy to define in practice. On all our 29 benchmark tasks, they can be written in one line of code (see Appendix B), which is much shorter and simpler than the human-written shaped reward functions that we compare Eureka to. Therefore, we believe that “tasks that come with known ground-truth reward functions” are not difficult to satisfy in practice, and Eureka readily applies to all such tasks with programmatic success conditions. This is a large space of tasks, and we have extensively evaluated a broad spectrum of robotic tasks in our experiments. Therefore, we believe that our approach’s applicability is already broad.

Furthermore, we do even show results where no fitness function is analytically expressed. In Section 4.4, we have studied cases where the desired behaviors are difficult to express via a mathematical function, such as running in a stable gait. In these cases, we have introduced Eureka from Human Feedback, which allows humans to express the “fitness” feedback in textual form, bypassing an analytical fitness function. We have shown that Eureka is indeed compatible with such a notion of fitness and can effectively generate reward functions that better align with human intention. In our limitation section in the updated manuscript (Section 5), we have also discussed the possibility of using a vision-language model (VLM) to construct automated textual feedback to improve the reward functions, providing a scalable alternative to human-based fitness feedback.

Please let us know whether our response has sufficiently addressed your concern about the fitness function; we are happy to provide more clarification as we believe that this is an important point that we’d like the reviewer to reach an agreement with.

Reviewer comment (Reproducibility):
The paper does not detail how the pre-trained policy was obtained. In the statement, "Specifically, we first instruct EUREKA to generate a reward for ... Then, using this pre-trained policy," it is unclear what exactly "this" references to. The application of curriculum learning here appears to break the policy training into two main stages — an initial 'pre-training' stage and a 'fine-tuning' stage with the pre-trained policy. If this interpretation is accurate, clarity around the specific process is crucial.

Author response:
The pre-training stage is identical to the original Shadow Hand environment in the Isaac Gym benchmark suite we study, except that we swapped out the cube object to the pen object. The application of Eureka for this stage is identical to the benchmark tasks, in which Eureka iteratively discovers better reward functions. The best Eureka reward is then used to train a policy to solve the task; the resulting converged policy from this training is the “pre-trained” policy.

Reviewer comment (Evaluation):
It is unclear whether the training process — both pre-training and fine-tuning stages — is guided by the reward functions derived using the LLM-powered reward design method proposed in this paper, and whether these reward functions are identical. If they are identical, there needs to be a detailed explanation of the distinctiveness of 'pre-training' and 'fine-tuning' in optimizing policies under the same reward function, and an explanation of how a dual-stage policy optimization can notably enhance policy performance. If this is the case, it appears that the success of the task may be due to the dual-stage policy optimization rather than the reward design, casting doubts on the paper’s central argument.

Author response:
Yes, during both stages, the policies are trained using the best Eureka-discovered reward function from the pre-training stages. These reward functions are hence identical.