Paper: Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
The paper essentially proposes prompting techniques for code generation with pre-trained language models accessible via API, specifically GPT-4. Despite its interesting conclusion and results, it consists at most of an interesting observational study over the capabilities of GPT-4, as the prompting techniques appear straightforward, fundamentally easy to execute manually by any user of the GPT-4 online API platform in a process of trial and error for reward design. While similar “prompting techniques” results have been published before in machine learning venues (such as the famous “chain of thought” prompting technique), this limits the significance of the work, as GPT-4 simply fills in the shoes of an expert reinforcement learning reward coder on well-known simulation environments, whereas “chain of thought” prompting was an innovation in how to query a model for more general-purpose NLP benchmarks.

Author response:
### **W1: The significance of our work**
**A:** We appreciate the reviewer's perspective. However, we respectfully disagree it is limited to the significance of the method this work proposed. To clarify, we want to emphasize the promise of this work for different communities (RL, Robotics, and NLP) as follows:

For the **reinforcement learning community**, it is promising to use LLMs to automate policy learning by generating reward functions that are difficult to write down by hand. It is not only an important application scenario for replacing human experts in writing rewards functions but also a novel way that has never been shown before to pour LLMs’ knowledge and abilities into the RL for robotics and control, as well as games. Our work shows promising initial results for further exploration in the future (as shown in the main experiments of Figure 2). 

For the **robotics community**, one of the most important things in the field of robotics is how to learn low-level motor skills at large scales efficiently, scalably, and automatically (or in a self-supervised manner) [1][2]. We believe that by leveraging LLMs, our method (automatic text-to-reward) takes a solid step toward automatically learning large-scale motor skills.

For the **natural language processing community**, our work proposes text-to-reward as a hard task for code generation and shows a huge gap in performance between the most advanced model and ones from the open-source community (as shown in the newly added experiments on Llama-2 and Code-Llama in Appendix F), showing the potential of improving in this aspect for people in the NLP community. 

Overall, our method could be an important direction in the future as a novel way to pour the knowledge learned from LLMs into general RL, low-level robotics skills learning and control, which are not the focus of previous studies that focus on high-level planning, and the method could get more effective with the growth of LLMs.

[1] Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Zackory Erickson, David Held, Chuang Gan, RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation https://arxiv.org/abs/2311.01455 

[2] Open X-Embodiment: Robotic Learning Datasets and RT-X Models, https://arxiv.org/abs/2310.08864

Reviewer comment (Reproducibility):
The reliance of the paper on GPT-4 makes the results inherently irreproducible, as OpenAI does not have a policy of indefinitely supporting API access to specific snapshot versions of their models (specifically, GPT-4-0314 will be deprecated on June 13th 2024 at the earliest).

Author response:
### **W1 & Q2: Tests with open source or publicly available models for reproducibility.**
**A:** Thanks for your suggestion! As you mentioned, it is a common practice to use the advanced GPT models from OpenAI or close source model PaLM from Google for novel applications and studies in previous work (Chain-of-Thought[1], Zero-shot planner[2], SayCan[3], ReAct[4]). For the reproducibility, we used open-source algorithms and environments and provided full reproducible prompts and code. 

Following your suggestion, we also add experiments in the application of our method using open-source or publicly available models on Llama-2 and Code-Llama (Llama-2 further pretrained on code corpus) on 10 tasks of MetaWorld. We include these findings in our revised manuscript and provide a more comprehensive evaluation of our method's capabilities across different models in Appendix F. The results show that the open-source models still have a huge gap with the most advanced models in the difficult task of reward function generation and reflect the necessity of using the most advanced one to demonstrate the possibility of our method.

[1] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.
https://arxiv.org/abs/2201.11903

[2] Wenlong Huang, Pieter Abbeel, Deepak Pathak, Igor Mordatch. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. 
https://arxiv.org/abs/2201.07207

[3] Michael Ahn, Anthony Brohan, Noah Brown, et al,. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
https://arxiv.org/abs/2204.01691

[4] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models.
https://arxiv.org/abs/2210.03629

Reviewer comment (Experiments):
Would it be possible to show how this method generalizes to environments that are sure not to have been included in the training data of the model? (e.g., a simple simulated environment designed specifically for this paper)

Author response:
### **W1 & Q1: Generalization to environments not included in GPT-4's training data.**
**A:** Thanks for your question! We understand your concern regarding the potential data leakage of environment and reward code in GPT-4 and other LLMs, which might lead one to assume that the model's ability to generate high-quality code stems from exposure to "golden environments" and "golden examples."

However, we would like to clarify **three sources of evidence** from our paper and previous work:
1. Evidence from the input side (environment and prompts) and output side (generated reward function) of our proposed method for reward function generation;
2. Evidence from the locomotion novel skills experiments (including backflip, frontflip, wave legs) part in our paper;
3. Evidence from previous research showed the generalization capability on code generation problems of new domains that are not included in the training set.

**1. Evidence from the input side (prompt) and output side (generated reward function) of our proposed method for reward function generation.**

As much as you suggest using a simple, purpose-designed simulated environment, we would like to clarify that the Python classes and properties for our prompts were crafted by our team and differ significantly from the environment's original code. We then map the generated code to the environment's original code using a one-to-one deterministic mapping.

**From the input end,** one piece of evidence, as shown in Appendix C, is that the prompts we provided to LLMs like GPT-4 for our T2R method are **derived from expertly abstracted class Python representations, with specific attributes and function names coined by us, not exact snippets from the ManiSkill2 or MetaWorld GitHub repositories**. This rules out the possibility of the language model plagiarizing directly through context memory since previous work [1] has attempted to perturb instructions used for code generation and found significant performance drops. 

On the other hand, we also have preliminary results that entering direct commands like "generate a reward function for crawler" or "pick the cube" into GPT-4, or inputting the ManiSkill2 environment code context into GPT-4 to let it complete the `compute_dense_reward` section, none of which were successful. This not only demonstrates the inadequacy of mere memorization but also validates the effectiveness of our method.

**From the output end,** another piece of evidence is shown in Appendix D.1 of our paper, where we compare instances of code generated by LLM through zero-shot and few-shot prompting with the oracle code example on the ManiSkill2 Pick Cube task.

Below is the code generated by zero-shot prompting GPT-4 using the T2R method:
```python
import numpy as np

def compute_dense_reward(self, action): 
    # Normalize action 
    action = np.clip(action, -1, 1) 

    # Calculate distance between gripper and cube 
    gripper_pos = self.robot.ee_pose.p 
    cube_pos = self.cubeA.pose.p 
    dist_gripper_cube = np.linalg.norm(gripper_pos - cube_pos) 

    # Calculate distance between cube and goal 
    goal_pos = self.goal_position dist_cube_goal = np.linalg.norm(goal_pos - cube_pos) 

    # Check if the robot is grasping the cube 
    grasping_cube = self.robot.check_grasp(self.cubeA) 

    # Define reward components 
    reward_dist_gripper_cube = -1.0 * dist_gripper_cube 
    reward_dist_cube_goal = -1.0 * dist_cube_goal 
    reward_grasping_cube = 1.0 if grasping_cube else -1.0 

    # Define weights for reward components 
    weight_dist_gripper_cube = 0.3 
    weight_dist_cube_goal = 0.5 
    weight_grasping_cube = 0.2 

    # Calculate total reward 
    reward = weight_dist_gripper_cube * reward_dist_gripper_cube \ 
                + weight_dist_cube_goal * reward_dist_cube_goal \ 
                + weight_grasping_cube * reward_grasping_cube 

    # Regularization on action 
    reward -= 0.01 * (action ** 2).sum() 
    return reward
```

And here is the oracle code (from the experts of the ManiSkill team):
```
import numpy as np

def compute_dense_reward(self, action):
    reward = 0.0

    is_obj_placed = np.linalg.norm(self.goal_position - self.cubeA.pose.p) <= 0.025
    is_robot_static = np.max(np.abs(self.robot.qvel)) <= 0.2

    success = is_obj_placed and is_robot_static

    if success:
        reward += 5
        return reward

    tcp_to_obj_pos = self.cubeA.pose.p - self.robot.ee_pose.p
    tcp_to_obj_dist = np.linalg.norm(tcp_to_obj_pos)
    reaching_reward = 1 - np.tanh(5 * tcp_to_obj_dist)
    reward += reaching_reward

    is_grasped = self.robot.check_grasp(self.cubeA, max_angle=30)
    reward += 1 if is_grasped else 0.0

    if is_grasped:
        obj_to_goal_dist = np.linalg.norm(self.goal_position - self.cubeA.pose.p)
        place_reward = 1 - np.tanh(5 * obj_to_goal_dist)
        reward += place_reward

    return reward
```

If the process were reliant on memory replication, these two code segments would be very similar. However, here the styles and approaches of the reward are quite different, yet they ultimately achieve the same effect. Thus, we wouldn’t say the "leakage" argument negates the efficacy of the T2R method.

**2. Evidence from the locomotion novel skills experiments (including backflip, frontflip, wave legs) part in our paper**

We did experiments on learning novel skills in MuJoCo, as shown in Table 1 and Figure 4, T2R generates reward codes under our created new instructions for novel skills. We also show the generated reward function codes in Appendix D.2.

**3. Evidence from previous research showed the generalization capability on code generation problems of new domains that are not included in the training set**

Previous research has shown the generalization capability and great performance on code generation problems that are not included in the training set of GPT-4 by conducting:

 - contamination study [2]: According to Appendix C (Contamination on professional and academic exams), Appendix D (Contamination on academic benchmarks), Table 9, Table 10, and Table 11, the GPT-4 team from OpenAI trained different versions of GPT-4 w. and w.o. the content from the evaluation set and conclude “contamination overall has very little effect on the reported results.”.
 - testing on fresh coding questions after GPT-4’s pretraining period [3]. They construct a benchmark of 100 LeetCode problems posted after October 8th, 2022, which is after GPT-4’s pretraining period and show that GPT-4 significantly outperforms the other models and is comparable to human performance on these fresh questions. 

[1] Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen-tau Yih, Daniel Fried, Sida Wang, Tao Yu. DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation https://arxiv.org/abs/2211.11501

[2] OpenAI, GPT-4 Technical Report
https://arxiv.org/abs/2303.08774 

[3] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, Yi Zhang. Sparks of Artificial General Intelligence: Early experiments with GPT-4
https://arxiv.org/abs/2303.12712

Reviewer comment (Reproducibility):
Did you try your method on open source or otherwise publicly available models that would allow for a snapshot of the results to be preserved? If so, how do they compare to GPT-4?

Author response:
### **W1 & Q2: Tests with open source or publicly available models for reproducibility.**
**A:** Thanks for your suggestion! As you mentioned, it is a common practice to use the advanced GPT models from OpenAI or close source model PaLM from Google for novel applications and studies in previous work (Chain-of-Thought[1], Zero-shot planner[2], SayCan[3], ReAct[4]). For the reproducibility, we used open-source algorithms and environments and provided full reproducible prompts and code. 

Following your suggestion, we also add experiments in the application of our method using open-source or publicly available models on Llama-2 and Code-Llama (Llama-2 further pretrained on code corpus) on 10 tasks of MetaWorld. We include these findings in our revised manuscript and provide a more comprehensive evaluation of our method's capabilities across different models in Appendix F. The results show that the open-source models still have a huge gap with the most advanced models in the difficult task of reward function generation and reflect the necessity of using the most advanced one to demonstrate the possibility of our method.

[1] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.
https://arxiv.org/abs/2201.11903

[2] Wenlong Huang, Pieter Abbeel, Deepak Pathak, Igor Mordatch. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. 
https://arxiv.org/abs/2201.07207

[3] Michael Ahn, Anthony Brohan, Noah Brown, et al,. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
https://arxiv.org/abs/2204.01691

[4] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models.
https://arxiv.org/abs/2210.03629

Reviewer comment (Experiments):
The authors should compare against Yu 2023, especially if claiming their dense free-form reward + use of established coding packages can result in superior performance. The comparison isn’t exactly 1-1 given the claims, but currently there is no comparison to any baseline to contextualize the performance of the method. In fact, I think a comparison with Yu 2023 + RL would be fairest, as Yu 2023 likely can use RL instead of MPC without change.

Author response:
2. **L2R can only use a sum of reward terms v.s. T2R can write stage and dense rewards such as if-else statements:**
   1. Our use of dense rewards, as opposed to sparse rewards, allows for a more nuanced and continuous evaluation of agent behaviors, crucial for complex tasks that involve many steps.
   2. For example, the stack cube task needs "pick up cubeA, place it on cubeB, release cubeA and make sure cubeA is static after releasing". However, L2R will lead to a contradiction between the reward term that encourages "pick up cubeA" and "release cubeA" when writing them in a simple summation expression.
   3. To show the necessity of dense rewards, we trained RL with oracle-sparse-reward on ManiSkill2 as an additional experiment in Appendix E. This baseline adapts the oracle expert-written reward codes to mirror the L2R's format (the sum of a set of individual terms) by removing all if-else statements and only keeping functional reward terms, while disregarding the original L2R's inability to utilize point clouds for distance calculations. While "oracle-sparse-reward + RL" yielded results comparable to the zero-shot setting for 3 simpler tasks, it completely can not work (i.e. success rate ≈ 0) on the remaining 3 more challenging tasks, underscoring the limitations of sparse rewards.
3. **L2R is model-based and needs additional expert effort v.s. T2R is model-free and can easily be transferred to new environments and real-world application:**
   1. The efficacy of the L2R approach is significantly contingent on the performance of Model Predictive Control (MPC). While it is a well-accepted notion that model-based methods, like MPC, outperform model-free methods when an accurate world model is accessible, the practical deployment of MPC is constrained by the requirement for such a model. Crafting this world model demands additional expertise and is not straightforwardly translatable to real-world applications due to this dependency.
   2. The selection of reward term, design reward term template, and even finding appropriate reward term weight all needs an extra large amount of expert effort.

Reviewer comment (Writing):
4.1: Appendex → Appendix.

Author response:
**A:** We appreciate your suggestions. The typographical errors have been corrected in our revised manuscript. While we acknowledge the value of including reward function examples for clarity, space constraints necessitate a concise presentation. Therefore, only a key example is showcased in Figure 1. We intend to include more comprehensive examples in the camera-ready version, where space permits.

Reviewer comment (Presentation):
I think it’d be nice to have a few small examples in the main paper of generated reward functions (not full things, just a few lines). This makes the experiment section more readable without needing to jump around to the appendix.

Author response:
**A:** We appreciate your suggestions. The typographical errors have been corrected in our revised manuscript. While we acknowledge the value of including reward function examples for clarity, space constraints necessitate a concise presentation. Therefore, only a key example is showcased in Figure 1. We intend to include more comprehensive examples in the camera-ready version, where space permits.

Reviewer comment (Novelty):
Eureka has recently come out (https://eureka-research.github.io/), and as concurrent work, what do the authors think they contribute compared to Eureka?

Author response:
**A:** Eureka represents an important concurrent development in the field, we all address the important problem of dense reward specification in reinforcement learning, using a method based on zero-shot and few-shot prompting of a language model, as Reviewer#4 pointed out. We believe that our work, alongside Eureka, contributes to the broader understanding and capabilities of LLMs in policy training and reinforcement learning.

Reviewer comment (Experiments):
The primary weakness of the paper is that most evaluation tasks are from benchmarks that have been released before GPT-4's knowledge cutoff date (September, 2021). Mujoco and Metaworld tasks have been extensively studied in the reinforcement learning literature; ManiSkill2, though released recently, have many overlapping tasks with ManiSkill, which was released in mid 2021; in particular, most of the tasks, to the best of my knowledge, were in the original ManiSkill benchmark. Given this, it is not clear whether the reward design capability of T2R can readily transfer to an unseen task in a new simulator.

Author response:
### **W 1,2,3: Data leakage in GPT-4**
**A:** Thanks for your question! We understand your concern regarding the potential data leakage of reward code in GPT-4 and other LLMs, which might lead one to assume that the model's ability to generate high-quality code stems from exposure to "golden examples."

However, we would like to clarify three sources of evidence from our paper and previous work:
1. Evidence from the locomotion novel skills experiments (including backflip, frontflip, wave legs) part in our paper;
2. Evidence from the input side (prompt) and output side (generated reward function) of our proposed method for reward function generation;
3. Evidence from previous research showed the generalization capability on code generation problems of new domains that are not included in the training set.

**1. Evidence from the locomotion novel skills experiments (including backflip, frontflip, wave legs) part in our paper**

We did experiments on learning novel skills in MuJoCo, as shown in Table 1 and Figure 4, T2R generates reward codes under our created new instructions for novel skills. We also show the generated reward function codes in Appendix D.2.

And for your concern about Christiano et al., 2017, as far as we know, although a hopper backflip was demonstrated in their work, this behavior is learned from human preferences using a neural network reward model, rather than using a symbolic reward code. Furthermore, our generated reward function (shown in Appendix D.2) is very different from what they showed as an expert-written case in their blog (https://openai.com/research/learning-from-human-preferences) of this work.

**2. Evidence from the input side (prompt) and output side (generated reward function) of our proposed method for reward function generation.**

**From the input end,** one piece of evidence, as shown in Appendix C, is that the prompts we provided to LLMs like GPT-4 for our T2R method are **derived from expertly abstracted class Python representations, with specific attributes and function names coined by us, not exact snippets from the ManiSkill2 or MetaWorld GitHub repositories**. This rules out the possibility of the language model plagiarizing directly through context memory since previous work [1] has attempted to perturb instructions used for code generation and found significant performance drops. 

On the other hand, we also have preliminary results that entering direct commands like "generate a reward function for crawler" or "pick the cube" into GPT-4, or inputting the ManiSkill2 environment code context into GPT-4 to let it complete the `compute_dense_reward` section, none of which were successful. This not only demonstrates the inadequacy of mere memorization but also validates the effectiveness of our method.

Reviewer comment (Novelty):
Relatedly, the "novel" behavior on the Mujoco locomtoin tasks have appeared in prior literature; for example, Hopper back flip is shown in Christiano et al., 2017. It's unclear whether T2R has benefited from that knowledge possibly being in the training set of the backbone LLM.

Author response:
### **W 1,2,3: Data leakage in GPT-4**
**A:** Thanks for your question! We understand your concern regarding the potential data leakage of reward code in GPT-4 and other LLMs, which might lead one to assume that the model's ability to generate high-quality code stems from exposure to "golden examples."

However, we would like to clarify three sources of evidence from our paper and previous work:
1. Evidence from the locomotion novel skills experiments (including backflip, frontflip, wave legs) part in our paper;
2. Evidence from the input side (prompt) and output side (generated reward function) of our proposed method for reward function generation;
3. Evidence from previous research showed the generalization capability on code generation problems of new domains that are not included in the training set.

**1. Evidence from the locomotion novel skills experiments (including backflip, frontflip, wave legs) part in our paper**

We did experiments on learning novel skills in MuJoCo, as shown in Table 1 and Figure 4, T2R generates reward codes under our created new instructions for novel skills. We also show the generated reward function codes in Appendix D.2.

And for your concern about Christiano et al., 2017, as far as we know, although a hopper backflip was demonstrated in their work, this behavior is learned from human preferences using a neural network reward model, rather than using a symbolic reward code. Furthermore, our generated reward function (shown in Appendix D.2) is very different from what they showed as an expert-written case in their blog (https://openai.com/research/learning-from-human-preferences) of this work.

**2. Evidence from the input side (prompt) and output side (generated reward function) of our proposed method for reward function generation.**

**From the input end,** one piece of evidence, as shown in Appendix C, is that the prompts we provided to LLMs like GPT-4 for our T2R method are **derived from expertly abstracted class Python representations, with specific attributes and function names coined by us, not exact snippets from the ManiSkill2 or MetaWorld GitHub repositories**. This rules out the possibility of the language model plagiarizing directly through context memory since previous work [1] has attempted to perturb instructions used for code generation and found significant performance drops. 

On the other hand, we also have preliminary results that entering direct commands like "generate a reward function for crawler" or "pick the cube" into GPT-4, or inputting the ManiSkill2 environment code context into GPT-4 to let it complete the `compute_dense_reward` section, none of which were successful. This not only demonstrates the inadequacy of mere memorization but also validates the effectiveness of our method.

Reviewer comment (Experiments):
Most manipulation tasks are of "pick-and-place" or opening/closing/pushing nature. These are also the most common types of manipulation tasks that the RL literature has studied. It is possible that GPT-4 is generally adept at writing reward functions for those task types.

Author response:
### **W 1,2,3: Data leakage in GPT-4**
**A:** Thanks for your question! We understand your concern regarding the potential data leakage of reward code in GPT-4 and other LLMs, which might lead one to assume that the model's ability to generate high-quality code stems from exposure to "golden examples."

However, we would like to clarify three sources of evidence from our paper and previous work:
1. Evidence from the locomotion novel skills experiments (including backflip, frontflip, wave legs) part in our paper;
2. Evidence from the input side (prompt) and output side (generated reward function) of our proposed method for reward function generation;
3. Evidence from previous research showed the generalization capability on code generation problems of new domains that are not included in the training set.

**1. Evidence from the locomotion novel skills experiments (including backflip, frontflip, wave legs) part in our paper**

We did experiments on learning novel skills in MuJoCo, as shown in Table 1 and Figure 4, T2R generates reward codes under our created new instructions for novel skills. We also show the generated reward function codes in Appendix D.2.

And for your concern about Christiano et al., 2017, as far as we know, although a hopper backflip was demonstrated in their work, this behavior is learned from human preferences using a neural network reward model, rather than using a symbolic reward code. Furthermore, our generated reward function (shown in Appendix D.2) is very different from what they showed as an expert-written case in their blog (https://openai.com/research/learning-from-human-preferences) of this work.

**2. Evidence from the input side (prompt) and output side (generated reward function) of our proposed method for reward function generation.**

**From the input end,** one piece of evidence, as shown in Appendix C, is that the prompts we provided to LLMs like GPT-4 for our T2R method are **derived from expertly abstracted class Python representations, with specific attributes and function names coined by us, not exact snippets from the ManiSkill2 or MetaWorld GitHub repositories**. This rules out the possibility of the language model plagiarizing directly through context memory since previous work [1] has attempted to perturb instructions used for code generation and found significant performance drops. 

On the other hand, we also have preliminary results that entering direct commands like "generate a reward function for crawler" or "pick the cube" into GPT-4, or inputting the ManiSkill2 environment code context into GPT-4 to let it complete the `compute_dense_reward` section, none of which were successful. This not only demonstrates the inadequacy of mere memorization but also validates the effectiveness of our method.

Reviewer comment (Evaluation):
T2R appears to still work best with few-shot examples. In many tasks that do not belong to a family of tasks introduced by a benchmark, providing few-shot examples can still be difficult.

Author response:
### **W4: Consideration of the difficulty when using few-shot T2R in new domain**
**A:** Thank you for the inquiry. We want to clarify that few-shot examples, while beneficial, are not a prerequisite for the success of our T2R approach, given the current strength of LLMs. Our MetaWorld zero-shot setting experiments demonstrate that the rewards generated are on par with those crafted by human experts and are effective for developing novel locomotion skills. We included few-shot setting experiments as a means to enhance performance, aligning with established practices in in-context learning and code generation research [1, 2, 3].

[1] Tom B. Brown, Benjamin Mann, Nick Ryder, et al., Language Models are Few-Shot Learners
https://arxiv.org/abs/2005.14165

[2] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa, Large Language Models are Zero-Shot Reasoners
https://arxiv.org/abs/2205.11916

[3] Mark Chen, Jerry Tworek, Heewoo Jun, et al., Evaluating Large Language Models Trained on Code
https://arxiv.org/abs/2107.03374

Reviewer comment (Experiments):
For each task, only one reward function is reported. It is not clear whether T2R is robust to stochasticity in LLM generation.

Author response:
### **W5: Not clear whether T2R is robust to stochasticity in LLM generation**
**A:** Thank you for raising this point. In our error analysis presented in Table 2, we examined 100 reward function examples generated by both zero-shot and few-shot prompts across 10 different Maniskill2 tasks. This involved a detailed review of 10 unique reward codes generated by a language model per task with the same prompt to assess accuracy. While policy training for success rate stability could offer deeper robustness insights, our current aim is to showcase T2R's capability to simplify policy learning. We'll consider adding robustness experiments in the camera-ready version.

Reviewer comment (Experiments):
There are 11 tasks in MetaWorld, but only 6 of them have results in the paper?

Author response:
### **Q1: Why are only 6 out of the 11 tasks in MetaWorld reported in the paper?**
**A:**  To clarify, we include 10 tasks from Metaworld in the same fashion as follows [1] and [2]. Among all 17 included tasks, 7 are from Maniskill and 10 (instead of 11) are from Metaworld. The 7 figures for Maniskill tasks are 6 subfigures in Figure 2 and 1 figure in Figure 7 (to avoid replotting the same result twice). The 10 figures for Metaworld are 6 subfigures in Figure 3 and 4 subfigures in Figure 8 in the Appendix (to save space in the main text). 

[1] Kimin Lee, Laura Smith, Pieter Abbeel. PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training https://arxiv.org/abs/2106.05091 

[2] Joey Hejna, Dorsa Sadigh. Few-Shot Preference Learning for Human-in-the-Loop RL https://arxiv.org/abs/2212.03363

Reviewer comment (Experiments):
Is there a reason few-shot examples are not used for MetaWorld?

Author response:
### **Q2: Is there a specific reason why few-shot examples were not used for MetaWorld?**
**A:** In MetaWorld, given its relatively straightforward task set with simple reward function expressions, we found that most challenges could be addressed through zero-shot prompting. This underscores the potential for generating effective rewards by relying solely on a compact representation of the environment.

Reviewer comment (Reproducibility):
How robust is T2R to different random generations? The temperature used for the LLM generation is not shown.

Author response:
### **Q3: How robust is T2R to different random generations? What temperature setting was used for LLM generation?**
**A:** Thank you for your question! In line with what we discussed in response to W5, we've indeed touched upon this in Table 2. We'd like to point out that while LLM generation robustness is interesting, our primary goal here is to illustrate how T2R can facilitate the simplification of policy learning. 
For GPT-4, we set the temperature to 0.7 for all the experiments.

Reviewer comment (Experiments):
Can T2R work for non pick-and-place, opening/closing/pushing robotic manipulation tasks?

Author response:
### **Q4: Can T2R work for non pick-and-place, opening/closing/pushing robotic manipulation tasks?**
**A:** In theory, our formulation can do any task if its reward can be written in the format of code. Technically, for 2-finger parallel gripper manipulation, we believe our method can successfully work for assembly tasks such as peg-in-the-hole. To apply our method to dexterous hand manipulation, as well as deformable object (soft-body, cloth, etc.) manipulation, one can extend our reward to compute the distance between meshes and fingers.

Reviewer comment (Evaluation):
There is no qualitative analysis/discussion of what the source of the improvement is: Why does Few-shot fail to outperform Oracle even though Oracle is in context (Lift Cube, Turn Faucet, Push Chair, Open Cabinet Drawer)?

Author response:
**W1.1 & W1.2: Why does zero-shot sometimes outperform few-shot? Why does few-shot fail to outperform oracle even though oracle is in context?**

**A:** Thanks for your question! We observed that the LM tends to copy code snippets from the few-shot code when provided. If the human oracle is not optimal, the few-shot human oracles might inadvertently introduce biases or constraints, limiting the LM's creative reward generation.

Reviewer comment (Evaluation):
There is no qualitative analysis/discussion of what the source of the improvement is: In the cases that few-shot improves on zero-shot, what is the source of this improvement?

Author response:
**W1.3: In the cases that few-shot improves on zero-shot, what is the source of this improvement?**

**A:** Robotic tasks are related, especially the dense stages rewards. When few-shot improves on zero-shot, the few-shot examples likely provide relevant, task-specific stage rewards based on which the LM can rewrite.

Reviewer comment (Experiments):
There are a lot of missing details in the experiments: There is only one example of human feedback (Figure 6) and it is in a schematic diagram for a task that does not have ground truth. I would like to see a few traces of the whole round (generation, feedback, generation, feedback, generation) in order to understand what exactly is happening.

Author response:
**W2.1: Clarification on the human feedback process and its trace in the experiments.**

**A:** In the revised manuscript Appendix D.3, we added a detailed example showing the traces of the whole round to show how human feedback improves the reward code step by step.

Reviewer comment (Experiments):
There are a lot of missing details in the experiments: The experiments in Table 1 and Figure 4 are supposedly conducted in a zero-shot setting (caption in Figure 4), yet Figure 6 gives a schematic for ambiguity resolution which would imply a few-shot result for novel behaviors, an experiment which I do not see in the paper and appendices.

Author response:
**W2.3: Clarification on the setting of Figure 6**

**A:** The locomotion example in Figure 6 is zero-shot + human feedback, but not few-shot (because we do not have a human oracle for all locomotion tasks).

Reviewer comment (Reproducibility):
There are a lot of missing details in the experiments: I do not see the choice of \( k \) for the number of neighbors that appear in the few-shot context.

Author response:
**W2.4: The choice of $k$ for the number of neighbors in few-shot context.**

**A:** We set $k$ as 1 in all experiments of few-shot settings.

Reviewer comment (Reproducibility):
There are a lot of missing details in the experiments: For generated rewards in Appendix D.1 on Pick Cube, the few-shot vs. oracle code is almost indistinguishable except for 2 constants (cube placed at goal reward, grasped reward). Given this difference is so small, it seems important to know what the human feedback was — are we just getting lucky?

Author response:
**W2.5 & W2.6 & W2.7: Few-shot and oracle are sometimes non-distinguishable. Few-shot sometimes underperform oracle. Is it always the case that the Oracle code for the task is put into the context for Few-shot?**

**A:** We design the experiment so that the oracle code for the task is never in the context or few-shot. For each task, we remove its oracle code from the retrieval pool to make sure that the LM does not cheat. This explains why few-shot is not necessarily better than oracle because it cannot copy the oracle. When few-shot and oracle are similar, one potential reason is that there is a similar task in the few-shot example, and the LM can adapt that oracle code to fit into this task.

Reviewer comment (Experiments):
There are a lot of missing details in the experiments: As before, given that the few-shot vs. oracle code is so close on Pick Cube (the only example we have to judge), why is it the case that the few-shot generation is underperforming oracle generation in other settings (Lift Cube, Turn Faucet, Open Cabinet Drawer, Push Chair)?

Author response:
**W2.5 & W2.6 & W2.7: Few-shot and oracle are sometimes non-distinguishable. Few-shot sometimes underperform oracle. Is it always the case that the Oracle code for the task is put into the context for Few-shot?**

**A:** We design the experiment so that the oracle code for the task is never in the context or few-shot. For each task, we remove its oracle code from the retrieval pool to make sure that the LM does not cheat. This explains why few-shot is not necessarily better than oracle because it cannot copy the oracle. When few-shot and oracle are similar, one potential reason is that there is a similar task in the few-shot example, and the LM can adapt that oracle code to fit into this task.

Reviewer comment (Experiments):
There are a lot of missing details in the experiments: Is it always the case that the Oracle code for the task is put into the context for Few-shot?

Author response:
**W2.5 & W2.6 & W2.7: Few-shot and oracle are sometimes non-distinguishable. Few-shot sometimes underperform oracle. Is it always the case that the Oracle code for the task is put into the context for Few-shot?**

**A:** We design the experiment so that the oracle code for the task is never in the context or few-shot. For each task, we remove its oracle code from the retrieval pool to make sure that the LM does not cheat. This explains why few-shot is not necessarily better than oracle because it cannot copy the oracle. When few-shot and oracle are similar, one potential reason is that there is a similar task in the few-shot example, and the LM can adapt that oracle code to fit into this task.

Reviewer comment (Presentation):
Section 4.2 is about code errors that occur before any RL happens, and this seems like a necessary filtering step, but having an example of the generation/feedback process is much more important in the main body than Table 2.

Author response:
### **W3: The importance of code errors section vs. generation/feedback process details.**

**A:** Thank you for your suggestion! We wish to clarify that when using GPT-4, syntax errors are generally not encountered, as indicated in Table 2, where direct generation has only a 10% error rate. Furthermore, we found that by feeding the error messages from the cases with syntax errors back to GPT-4, all such errors can be rectified. Given that it has become commonplace to use error messages to guide GPT-4 in correcting syntax issues, especially considering that reducing syntax error rates is not the focus of our paper, we choose to not add this as a section.

Reviewer comment (Evaluation):
It would be nice to include code examples for the novel behavior tasks to see what is happening. Given there is no baseline in this case, simply presenting a quantitative evaluation without any analysis is a little sparse.

Author response:
### **W5: Inclusion of code examples for novel behavior tasks.**

**A:** Just as you mentioned, to enhance clarity and understanding, we include reward code examples generated for novel behavior tasks in Appendix D.2. This will provide a clearer picture of how the LM approaches these tasks and the nature of the generated rewards.

Reviewer comment (Evaluation):
The reviewer is most interested in analysis on the source of the improvements in each of the tasks, and how the language model goes about creating reward for a novel task.

Author response:
**W1.1 & W1.2: Why does zero-shot sometimes outperform few-shot? Why does few-shot fail to outperform oracle even though oracle is in context?**

**A:** Thanks for your question! We observed that the LM tends to copy code snippets from the few-shot code when provided. If the human oracle is not optimal, the few-shot human oracles might inadvertently introduce biases or constraints, limiting the LM's creative reward generation. 

**W1.3: In the cases that few-shot improves on zero-shot, what is the source of this improvement?**

**A:** Robotic tasks are related, especially the dense stages rewards. When few-shot improves on zero-shot, the few-shot examples likely provide relevant, task-specific stage rewards based on which the LM can rewrite.