Paper: Learning variable-length skills through Novelty-based Decision Point Identification (ICLR-2024)

Reviewer comment (Theory):
The assumption that "novel state-actions are critical decision points" is **too strong**. I find that datasets used in this paper just meet this assumption: in Maze, the behavior policy only chooses diverse actions at crossroads; in Kitchen, the dataset consists of expert policies accomplishing a fixed set of skills, making the actions diverse at skill success points only. If we use other environments (e.g., robotic tasks without clearly defined skills) or non-expert data collection policies with stochasticity, this assumption will no longer hold, and it is doubtful whether NBDI can outperform SPiRL.

Author response:
Thank you for your insightful feedback. In the sparse block stacking environment, the agent needs to stack blocks on top of each other. Since the agent’s sole objective is to stack as many blocks as possible, this environment can be regarded as a robotic task without clearly defined skills (whereas kitchen has clearly defined subtasks). Please refer to Appendix B for more details. 

Table 1 shows that NBDI outperforms SPiRL in the sparse block stacking environment (robotic tasks without clearly defined skills) with an improvement rate of 67.16%. Note it is a higher improvement rate compared to robotic task with clearly defined skills (kitchen: 22.33%). The significance of learning variable length skills with state-action novelty increases in environments lacking clearly defined subtasks (sparse block stacking).

Reviewer comment (Presentation):
According to the context, the presentation of the Theorem in Section 4.2 seems redundant. The theorem tells that breaking skills into shorter pieces can improve the high-level controller. But the paper studies how to determine the skill termination points, instead of breaking pre-defined fixed-length skills into shorter pieces. Also, the conclusion in the theorem is straight-forward, while the detailed notations (Q and V) in it are no longer used in the paper. I think the theorem should be moved to the Appendix.

Author response:
SPiRL basically extracts random fixed-length action sequences from the offline dataset by regarding the fixed-length action sequence in the dataset as pre-defined skills. On the other hand, our approach tries to break long action sequences using state-action novelty to learn better defined skills. Given that we are able to learn optimal skill (high-level) policy, the optimal way of defining skill would be using low-level actions directly as skills. In practice, however, shorter skills will increase the difficulty of learning, leading to poor skill policy. This is why we need a more clever and efficient break down of skills that resembles terminations with optimal improvement as shown in the Theorem [3] and the Figure 4, while obtaining similar temporal abstraction.

Reviewer comment (Novelty):
The method is incremental to SPiRL. Its improvement on the two robotic simulation tasks is small, according to Figure 6.

Author response:
Thank you for pointing out this. We've incorporated Table 1 to highlight the degree of quantitative improvement, especially in complex environments, compared to SPiRL[2]. Please refer to the general response to see the details.

Reviewer comment (Experiments):
As discussed in Weakness (1), when the behavior policies in the dataset are non-expert and stochastic, it is questionable whether NBDI can still provide benefits. More experimental results are required; otherwise, the authors should make this assumption and limitation clear in the paper.

Author response:
Thank you for your constructive advice. We've added a new section (Appendix C) to investigate the extent to which our proposed method depends on the quality of the offline dataset. Please refer to the general response to see the details.

Reviewer comment (Experiments):
In the experiments, how was the model capacity and the number of training steps of ICM adjusted? If ICM overfits on training data, can it still represent state-action novelty, as discussed in Weakness (2)?

Author response:
Thank you for raising this important point. We have included a section (Appendix E) to show that the estimated state-action novelty is not much influenced by the number of parameters or dataset used to train the model. We also doubled the number of training steps, but observed that the scale didn't differ significantly.

Reviewer comment (Reproducibility):
When generating termination labels for the dataset, how is the continuous prediction loss of ICM converted into binary termination labels? Does this require selecting a threshold manually?

Author response:
Thank you for pointing out this. Empirically, we discovered that state-actions falling within the top 1% of the prediction error percentile serves well as critical decision points, and used the corresponding threshold to learn the termination distribution.

Reviewer comment (Experiments):
Lack of option discovery/learning baselines. As the authors also mentioned in the introduction section, option framework and its related algorithms typically learn a termination condition which functions as a tool to determine whether the agent needs to switch to a different skill. Several recent papers [1, 2, 3] propose to do skill/option discovery from offline data while learning a flexible termination condition just like this paper. But none of them are compared in the empirical results or discussed in the related work section. The proposed approach is only compared to SPiRL which uses a fixed skill length.

Author response:
We appreciate your constructive advice for the experiment. We added a section (Appendix F) to compare NBDI with different variable length algorithm. Please refer to Appendix F for more details.

We also attempted to incorporate [2] into the comparison; however, we were unable to replicate the results reported in the paper since the dataset used to train [2] was not made available. So we mainly compared our model with [3] in Appendix F.

Reviewer comment (Experiments):
As it's a offline RL test setting, I think standard offline RL algorithms should be compared to instead of SAC. Moreover, as a general skill discovery method, I would suggest the authors test the proposed approach on more domains.

Author response:
As depicted in Figure 5, both NBDI and SPiRL[1] need environment interactions to train the skill policy. As our paper primarily focuses on the online RL setting, we believe SAC serves as a suitable baseline for comparison across various environments.

Reviewer comment (Experiments):
The algorithm is motivated with the concept of novelty, explicitly using a count-based notion of novelty in equation (1). In the experiments section the paper then briefly mentions that Intrinsic Curiosity Module (ICM) is used to obtain state-action novelty values. However, ICM obtains its intrinsic motivation signal from the prediction error of the next state in a learned feature space which is conceptually somewhat different. I think it would be a good idea to discuss to which extent the exact nature of the curiosity signal influences the learned skills.

Author response:
**1. To which extent the exact nature of the curiosity signal influences the learned skills.**

We acknowledge the concern raised regarding the missing details, and we would like to address this by providing the necessary information in Appendix B. Please refer to the general response to see the details.

Reviewer comment (Evaluation):
As evident in the motivating examples, the learned skill termination probabilities depend on the data, in particular on where different actions have been chosen frequently. This poses the question of how dependent NBDI is on a suitable structure being present in the offline dataset. For example, could NBDI work with exploratory data, and if yes, under which circumstances? How does the complexity of the environment influence the requirements on the data? A discussion of these questions would be a good addition to the paper.

Author response:
**2. How dependent NBDI is on a suitable structure being present in the offline dataset. For example, could NBDI work with exploratory data, and if yes, under which circumstances?**

We appreciate your constructive advice for the experiment. We've introduced a new section (Appendix C) to explore to what extent our proposed method is depend on the quality of the offline dataset. Please refer to the general response to see the details. 

**3. How does the complexity of the environment influence the requirements on the data?**

We have also added a section (Appendix D) to investigate whether meaningful decision points can be found in complex physics simulation environments. Please refer to Appendix D for more details.

Reviewer comment (Writing):
In the paragraph “Option Framework”, in the definition $\beta: \mathcal{S}^+ \rightarrow [0, 1]$ the symbol $\mathcal{S}^+$ is not defined anywhere. What does it stand for?

Author response:
**4. In the paragraph “Option Framework”, in the definition $\beta:S^+\to[0,1] $ the symbol  is not defined anywhere. What does it stand for?**

We apologize for the confusion caused. We initially intended to define $S^+$ as the termination set in accordance with [2], but we deemed it unnecessary and substituted it with the state space $S$.

Reviewer comment (Theory):
In equation (5) $\tilde{r}$ is added to the objective in each time step even though $\tilde{r}$ itself is already the cumulative discounted reward for the execution of one skill. Could you maybe explain how to reconcile these definitions?

Author response:
**5. In equation (5) $\tilde{r}$ is added to the objective in each time step even though $\tilde{r}$ itself is already the cumulative discounted reward for the execution of one skill. Could you maybe explain how to reconcile these definitions?**

Thank you for pointing this out. To reconcile the definition, we have now defined a set of time steps where we execute skills (Equation 5).

Reviewer comment (Reproducibility):
The method of variable length skills relies heavily on using state-action novelty to predict termination conditions. However, the exact details for how the state-action novelty module is trained are not explained in the paper (box (i) in the method figure). The paper includes some description of characterizing novelty as inverse visitation count, so do the authors explicitly maintain counts of each seen state? This will be difficult to scale to environments with continuous states and actions.

Author response:
Dear Reviewer 6rC9,

Thank you for the valuable feedback. We hope we can address your concerns below.

**1. Clarity of proposed approach (state-action novelty estimation)**

We apologize for the missing details. We've included a section (Appendix B) that details our application of the Intrinsic Curiosity Module (ICM)[1] in calculating state-action novelty. Please refer to the general response to see the details.

Reviewer comment (Novelty):
The proposed method heavily builds on prior work [1] for the components that learn skills and perform RL in the skill space. From the experimental results, the quantitative gains over [1] seem very marginal in the more complex environments (sparse block stacking, kitchen).

Author response:
Dear Reviewer 6rC9,

Thank you for the valuable feedback. We hope we can address your concerns below.

**2. The quantitive gains over SPiRL[2] seems very marginal in the more complex environments (sparse block stacking, kitchen).**

Thank you for pointing out this. We've included Table 1 to highlight the extent of quantitative improvement over SPiRL (including complex environments). Please refer to the general response to see the details.

Reviewer comment (Evaluation):
There is no analysis of how the discovered skills differ qualitatively in the more complex environments compared to [1]. What are the specific cases in these environments where variable length skills are actually helpful or needed?

Author response:
Dear Reviewer 6rC9,

Thank you for the valuable feedback. We hope we can address your concerns below.

**3. Can the authors include some analysis of how their discovered skills differ qualitatively in these more complex environments, as compared to [2]? What are the specific cases in these environments that variable length skills are actually helpful/needed?**

Thank you for the valuable suggestion. We've included Figure 11 (Appendix B) to illustrate states in the sparse block stacking environment that NBDI would learn to terminate (A, B, C) and states it would not learn to terminate (D, E, F). NBDI would learn to terminate in states where the agent has multitude of plausible actions (A, B, C) (e.g. choose to lift the block below or to move towards other blocks), while SPiRL would terminate in random states (A, B, C, D, E, F). As indicated by the improvement rate in Table 1, the significance of learning variable length skills with state-action novelty becomes increasingly crucial in environments lacking clearly defined subtasks (sparse block stacking), or in more complex settings (maze 40x40).