Paper: Proximal Curriculum with Task Correlations for Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Evaluation):
The metric for task similarity is a bit specific to the tasks. Currently, the metric is defined based on the L2 distance of context parameters. However, there are many cases where such a metric is not positively correlated with the intuitive similarity between tasks. For example, consider a table-top manipulation environment with a drawer and a cube on the table. Suppose the initial state is with the drawer closed and the cube on the table outside the drawer. Consider a desired state A with the cube on top of the closed drawer and a desired state B with the cube inside the closed drawer. Suppose the context parameters are defined as the open width of the drawer and the pose of the cube. We can see that the L2 distance between A and B is small, but the complexity of strategies to reach A and B is different: reaching state A only requires one pick-and-place while reaching state B requires opening the drawer, placing the cube, and then closing the drawer. It is inappropriate for the curriculum to treat these tasks as similar, and I think a better metric for task similarity is required.

Author response:
We thank the reviewer for pointing this out; it is indeed an interesting observation. In our work, we design our curriculum strategy to be agnostic to the task similarity metric, allowing for the application of any such metric. Nevertheless, for the purpose of the empirical evaluation, we opted to employ a standard similarity metric consistently across all domains. Exploring more generalized similarity metrics spanning various domains or even learning the similarity metric itself represents an interesting avenue for future research.

Reviewer comment (Evaluation):
Fig. 2(b) shows that PROXCORL performs similarly to PROXCORL-UN in PointMass-s:2G environment. Since the target task distribution is non-uniform in this environment, the result weakens the contribution of the proposed curriculum that takes task correlation into consideration.

Author response:
Based on the experimental results presented in the paper, we hypothesize that, for the PointMass-s:2G environment, the first two quantities (1) and (2) in Eq. (1), aligned with the ZPD principle, are enough to shift the training task distribution toward harder tasks. This observation aligns with similar trends identified in [1]. Through this shift towards harder tasks, the training task distribution ultimately converges to the target task distribution, even without explicit knowledge of the latter.

Conversely, in the PointMass-s:1T environment, where the target distribution is non-uniform, we observe that the performance gap between ProxCoRL and ProxCoRL-Un increases. Specifically, ProxCoRL converges more rapidly to the target distribution compared to ProxCoRL-Un. This indicates that, in the single target task setting, the incorporation of terms (3) and (4) intensifies the preference for tasks from the target distribution. This increased preference contributes to a more substantial distribution shift towards the target distribution, consequently improving the agent's speed in solving the target task.

Reviewer comment (Writing):
The above error occurs in following sections as well (see paragraph 3 in Section 3.1; First paragraph in Section 4).

Author response:
We thank the reviewer for these comments. We have updated the PDF with the suggested changes.

Reviewer comment (Presentation):
Minor: Figure 2 plotlines are a bit thick, making it somewhat difficult to read. I would suggest slightly decreasing the line width.

Author response:
We thank the reviewer for these comments. We have updated the PDF with the suggested changes.

Reviewer comment (Novelty):
The paper does not present a method that is fundamentally different from those in prior work [1]. The authors attempt to extend the idea proposed in ProCuRL [1] to a general target distribution, yet the core of this extension appears to be a mere application of an existing concept.

Author response:
The setting considered in the prior work [1] differs notably. Specifically, it does not account for a target task distribution, opting instead to gauge the agent's performance relative to a uniform distribution of tasks across the context space. In our work, the curriculum is developed with consideration for a target task setting, resulting in the introduction of additional terms (3) and (4) in Eq. (1). These terms enhance the preference for selecting tasks correlated with the target task.

Reviewer comment (Theory):
The simplified setting used to conduct the mathematical analysis diverges significantly from a typical RL setting, given it encompasses only two possible actions and a single isolated state space. This raises questions about the scalability of such an analysis in a general reinforcement learning framework.

Author response:
Our investigation into developing a curriculum strategy for the multi-task RL setting (outlined in Section 2) begins by observing that the following three essential factors must be taken into account when selecting a source-target task pair for the student component: (i) Learning potential in the source task, (ii) Transfer potential of the source task, i.e., similarity with the target task, and (iii) Potential performance gain in the target task. 

The challenge lies in integrating these quantities to devise an effective curriculum strategy. In pursuit of this goal, we opt to analyze a contextual bandit setting. Intriguingly, our analysis leads us to an intuitive curriculum strategy (Eq. (1)), involving the geometric mean of the three aforementioned factors. We extend this form to general learning settings with natural modifications. Furthermore, our empirical investigation demonstrates the effectiveness of our curriculum strategy.

In the revised version of this paper, we present a high-level approximation for the expected improvement in the training objective in a general RL setting (Section 3.1). Additionally, in the Appendix C.2, we conduct an extra analysis of a tree-structured contextual MDP setting. This analysis underscores the significance of the ZPD-related terms in our proposed curriculum strategy.

While it remains intriguing to delve into more complex learning settings and explore the potential for uncovering richer curriculum strategies, we acknowledge the general absence of such analyses in the existing literature.