Paper: Using Approximate Models for Efficient Exploration in Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
The novel components of the work are minor: training GNNs to model dynamics is well-studied (as referenced) and model-based RL is a well-studied field. The technique can be viewed as a form of exploration prior to favor exploitation of high value actions. Previous work on exploration techniques may be relevant, as these span a variety of ways to leverage approximate models for altering action selection (https://lilianweng.github.io/posts/2020-06-07-exploration-drl/ provides a survey). While those techniques are typically applied to modify rewards during learning, the core approximate models are closely related to the paper technique.

Author response:
Noted. Thanks for the resource on model-based exploration techniques which could serve as useful baselines. 
    CREATE is an environment designed to test generalisation to new actions in reinforcement learning. As such, the original paper 
    focuses on this problem (generalisation to new, unseen tools) and does not approach the tasks as we do (where the focus is on using 
    a single tool at training and test time). There are currently no SOTA baselines for CREATE for our context.

Reviewer comment (Experiments):
In absolute terms the success rate is low for goal success (ranging from ~0.2% to ~16%). Whether this is an improvement is not clear as the experiments only compare to a baseline of random action. No comparisons are made to SOTA on the task domain, nor are other models applied to this problem.

Author response:
Noted. Thanks for the resource on model-based exploration techniques which could serve as useful baselines. 
    CREATE is an environment designed to test generalisation to new actions in reinforcement learning. As such, the original paper 
    focuses on this problem (generalisation to new, unseen tools) and does not approach the tasks as we do (where the focus is on using 
    a single tool at training and test time). There are currently no SOTA baselines for CREATE for our context.

Reviewer comment (Experiments):
Results lack non-random baselines for comparison. This makes it difficult to assess how much the new technique improves over the state of the art.

Author response:
Noted. Thanks for the resource on model-based exploration techniques which could serve as useful baselines. 
    CREATE is an environment designed to test generalisation to new actions in reinforcement learning. As such, the original paper 
    focuses on this problem (generalisation to new, unseen tools) and does not approach the tasks as we do (where the focus is on using 
    a single tool at training and test time). There are currently no SOTA baselines for CREATE for our context.

Reviewer comment (Presentation):
Figure 1: What is the figure depicting?

Author response:
Figure 1 depicts how a dynamic ball interacts with a single static tool (for example, bucket tool) placed in the scene with the ball being launched at various angles and velocities at the tool.

Reviewer comment (Presentation):
Figure 2: This figure is probably not needed as the coordinate description is clear from the text.

Author response:
Noted.

Reviewer comment (Presentation):
Figure 3: What are the icons in this figure? What do these tasks illustrate as the task being accomplished?

Author response:
We will work on improving this in a future iteration of the paper. In Figure 3, the goal is to place tools in the scene so that the red ball 
    hits the green star goal object. The green circle represents a subgoal.

Reviewer comment (Presentation):
Figure 4: What do the icons in Task 1 and Task 2 mean compared to the colored blocks in the other tasks? What actions do these icons indicate? At what timestep(s) of the rollout?

Author response:
In Tasks 1 and 2, a bucket tool was used to solve the tasks, whereas, in the other tasks, a polygon tool was used. Sampling from the 
    action clusters over all time steps across multiple episodes results in the given distribution of actions.

Reviewer comment (Experiments):
Figure 5: Consider adding other baseline models to compare. Random is a reasonable lower bound, but there are no other points of comparison to judge how the current work advances over other methods.

Author response:
Noted. Thanks for the resource on model-based exploration techniques which could serve as useful baselines. 
    CREATE is an environment designed to test generalisation to new actions in reinforcement learning. As such, the original paper 
    focuses on this problem (generalisation to new, unseen tools) and does not approach the tasks as we do (where the focus is on using 
    a single tool at training and test time). There are currently no SOTA baselines for CREATE for our context.

Reviewer comment (Presentation):
Figure 6: What do the shaded areas represent?

Author response:
The shaded regions represent standard deviations where multiple seeds were run.

Reviewer comment (Experiments):
The description of the experiments needs improvement.

Author response:
We will work on improving this in a future iteration of the paper. In Figure 3, the goal is to place tools in the scene so that the red ball 
    hits the green star goal object. The green circle represents a subgoal.

Reviewer comment (Presentation):
Figures 2, 3 and 4 need more descriptive captions as well as better descriptions in the text. For example, what do the symbols correspond to in Fig 3? Similarly, in Fig 4, a legend (along with better descriptions) would have made it much easier to interpret the results.

Author response:
We will work on improving this in a future iteration of the paper. In Figure 3, the goal is to place tools in the scene so that the red ball 
    hits the green star goal object. The green circle represents a subgoal.

Reviewer comment (Evaluation):
Why is it that the relative Goal hits in Fig 5a are much higher compared to other tasks?

Author response:
I assume you are referring to Task 4 here. Task 4 may be an “easier” task in the sense that there is more leeway (or more options) in 
    the exact combination of actions that lead to the goal being achieved.

Reviewer comment (Experiments):
Over how many trials were the experiments conducted in Fig 6? In general, from the learning curves, the learning does not look stable.

Author response:
Learning was conducted over 2 trials due to time constraints. In a future iteration, we will increase this to improve the stability of 
    results.

Reviewer comment (Experiments):
Ablations showing the effect of $\epsilon_{threshold}$ are missing.

Author response:
Will include in a future iteration.

Reviewer comment (Theory):
As claimed in the last line on Page 7, why would the policy learning diverge? The belief is that learning would still occur (as DDPG learns off-policy) but due to a lack of exploitation, the learning curves would not reflect the learnt policy.

Author response:
Agreed.

Reviewer comment (Experiments):
The authors could have considered Phyre environments (https://ai.meta.com/tools/phyre/) to validate their approach.

Author response:
The issue with Phyre is that it is a one step environment, which limits its applicability for reinforcement learning. Do you have 
    other suggestions for domains to test our approach?

Reviewer comment (Presentation):
In Fig 6, it is assumed the orange color always corresponds to exploration epsilon of 0, but for task 6, the colors are swapped.

Author response:
Noted.

Reviewer comment (Writing):
The sentence “By using…however uncertain” towards the end of the introduction is too long and can be phrased better.

Author response:
Noted.

Reviewer comment (Experiments):
The second step of computing rewarding actions is a bit like sampling-based planning, for example, CEM or MPPI. Why do you use clustering here instead of CEM or MPPI?

Author response:
This is an interesting question. From our understanding, sampling-based planning does not focus on learning a policy but refines the 
    distribution from which candidate actions are sampled to then be evaluated under a model?
    Our aim was to directly assess how model inaccuracies can be mitigated through focussing on exploration priors to avoid suboptimal 
    policy learning. How do you propose we incorporate our exploration priors in a sampling-based planning setup? Or are you 
    proposing the sampling distribution acts as the exploration prior for policy learning?

Reviewer comment (Reproducibility):
How do you sample subgoals?

Author response:
The subgoals are indicated by the green circles in Figure 3. They indicate regions in space that the ball trajectory is required to pass 
    through in order to achieve the overall goal. They are hand-specified and we agree that this is a limitation. However, the focus of the 
    paper is not subgoal discovery, so perhaps a simpler heuristic algorithm to sample these subgoals would be a better approach?

Reviewer comment (Experiments):
If you first learn a model, why don’t you directly plan (for example, using MPC) in the world model, but using planning to guide the exploration, then learn a policy. Or directly do model-based reinforcement learning, since you already have a model. This combination, to me, would perform worse than planning and would be less efficient than MBRL. What’s the intuition for this combination?

Author response:
Please could you elaborate on this point. What is meant by planning here? Usually planning refers to the process of using a learned 
    model to learn a policy. Are you suggesting we use the policy learned using model-generated data to guide exploration in a model- 
    free setting? We did try this and the results were poor. Because of the model inaccuracy, using standard model-based RL results in 
    poor policy learning. 
    The intuition for using exploration priors is that, like humans, an agent can benefit from exploring in useful regions of the action space 
    to obtain more informative reward signals in a sparse reward task. Humans use approximate models to solve trial-and-error intuitive 
    physics tasks.

Reviewer comment (Experiments):
Task descriptions are not presented, what are these tasks in Fig.3, how do they work, where are subgoals, etc.

Author response:
We will work on improving this in a future iteration of the paper. In Figure 3, the goal is to place tools in the scene so that the red ball 
    hits the green star goal object. The green circle represents a subgoal.

Reviewer comment (Experiments):
A key weakness of the present submission is in the empirical evaluation. It is not clear if planning with the learned model (using the world model to train DDPG) would actually be problematic in this setting, which was posed as a vital motivating factor for this work.

Author response:
We did find that planning with the world model results in significantly deteriorated performance - we will include these results in a future iteration of the paper.

Reviewer comment (Experiments):
The paper only compares the proposed approach to standard DDPG. However, many approaches to model-based exploration use a learned model solely for exploration (see [1, 2]). For example, curiosity-driven approaches use a learned model to derive intrinsic rewards based on the model’s prediction error, information gain, etc. The paper would benefit from discussions and empirical comparisons to other model-based approaches to exploration.

Author response:
Your suggestion to include comparisons with other model-based exploration methods (for example, curiosity-driven approaches) is a very useful one and thanks for sharing the papers.