Paper: Subwords as Skills: Tokenization for Sparse-Reward Reinforcement Learning (ICLR-2024)

Reviewer comment (Presentation):
Some important results which do not speak in favor of the method are deferred to the appendices.

Author response:
Thank you for the encouraging words! We believe that there is a misunderstanding due to our poor presentation, and we hope the above clarifies this. We've changed Sections 4.1 and 4.3 to make these points more clear, including moving Appendix G into Section 4.3. Our original goal was to improve exploration, even in settings where demonstrations are not tightly aligned, so performance against methods with more assumptions was not the primary point.

Reviewer comment (Presentation):
The related work in the main paper could be more focused (see below).

Author response:
Thanks for this comment, we've now condensed this discussion in Section 4.1 and the related work, and grouped the baselines paragraph as suggested.

Reviewer comment (Presentation):
From the above 2 questions and remarks, my feeling is that to be fair, the main paper should have two tables like Table 1, one with learning from states and one with learning from images, where the separation of baselines and the environments all make sense. Eventually, if you don't have enough room, I think you could move all SSP results to an Appendix, as SSP is just an ablation of SPiRL. The fact that your method does not establish a new SOTA on all these results does not matter much, it has interesting enough features to deserve publication if properly presented.

Author response:
Thank you for the encouraging words! We believe that there is a misunderstanding due to our poor presentation, and we hope the above clarifies this. We've changed Sections 4.1 and 4.3 to make these points more clear, including moving Appendix G into Section 4.3. Our original goal was to improve exploration, even in settings where demonstrations are not tightly aligned, so performance against methods with more assumptions was not the primary point.

Reviewer comment (Evaluation):
In Table 3, SFP looks faster than SSP, whereas the paper seems to mention the contrary in several places. Can you clarify this?

Author response:
SFP is faster to train, but the online rollout is 4x slower. During RL, the rollout speed is what makes SFP painfully slow.

Reviewer comment (Experiments):
Is Table 3 obtained when learning from images or from states? In Table 1 (and/or 3 and/or 9), it would be nice to specify the number of obtained skills with your methods and the baselines. If this is a hyper-parameter, this should be made clear too.

Author response:
Thanks for these important clarifications. We used the same settings as our primary experiments in antmaze-medium-diverse, that is 16 skills with average length in Table 2 attempting to be consistent with length-10 skills of SSP. We detail most hyperparameters in Appendix E and reference this in the text. It appears we also missed the inclusion of default settings for N_min and N_max, which we add now in 4.1.

Reviewer comment (Reproducibility):
I did not find a list of hyper-parameters of your method. BTW, it would be nice to give a name to your method.

Author response:
Thanks for the catch! Please see our answer above on hyperparameters. We've also added a simple abbreviation (SaS for subwords as skills) to Section 3, which is used in all tables and figures.

Reviewer comment (Experiments):
Section 4.2 could be made clearer. At first read, it is unclear whether you are (1) using RL "from scratch", (2) you are learning from a dataset of demonstrations or (3) something intermediate. My guess is that up to the first half of the second paragraph, you speak about (1). But then, after "Moreover", you speak about weaknesses of SFP and SSP in the context of (2), without proper articulation of the ideas. And the remark about SAC goes back to (1). Again, Figure 5 displays "state visitation" of your method, but if your method is learning from a dataset, this makes no sense. All this must be clarified, including the title of the section.

Author response:
This is very helpful feedback. We make changes to the abstract at the introduction to clarify. We also added small paragraphs at the beginning of Section 3 and Section 4. We also retitle Section 4.1 and 4.3 to more directly address the problem setting.

Our method is
1. extract actions from reward-free demonstrations without any knowledge of when to use them'
2. do RL from scratch on a task with the modified action space

Hopefully these changes further clarify!

Reviewer comment (Presentation):
At the end of the paragraph in Section 4.2: "the large error bars of our method are a result of an optimization failure", we don't have pictures with error bars in the paper, just std info in some tables.

Author response:
Thanks for the catch, we've edited the wording to "standard deviations."

Reviewer comment (Presentation):
Figures 1 and 3 are not much informative, particularly Fig. 1b is not readable.

Author response:
We found it important to provide some visualization, though it's quite difficult to do so for Kitchen. We discuss this more in Appendix B, which we now reference in the caption.

Reviewer comment (Presentation):
The first paragraph of Section 4.1 is more about related work than about experiment. I think you should have a more focused related work section (mostly the "learning skills from demonstration" part) where you can give more details about these methods and reject what is less relevant into an Appendix. Another approach is to have a "Baselines" section where you describe the baselines with enough details so as to make the point you made in this paragraph.

Author response:
Thanks for this comment, we've now condensed this discussion in Section 4.1 and the related work, and grouped the baselines paragraph as suggested.

Reviewer comment (Presentation):
On page 9: results in Table 9 -> you must mention it is in an Appendix (but you should rather reorganize, see above).

Author response:
We've reorganized as mentioned.

Reviewer comment (Writing):
On page 4: "euclidean" should be "Euclidean".

Author response:
Nice catches, we've made changes.

Reviewer comment (Writing):
On page 8: "brownian" should be "Brownian".

Author response:
Nice catches, we've made changes.

Reviewer comment (Writing):
About paper writing: In Introduction, the background of RL and exploration is too long, while the study only focuses on the topic of skill discovery from data, which has been extensively studied. In Related Work, the section of "Skill Learning from Interaction" is also unnecessary because it is in parallel with your topic.

Author response:
Thank you for your comment! We wished to present the motivation from a broad perspective, as it is not always completely clear who the audience may be, and they may have different levels of experience with the subject matter, but we appreciate the precise perspective! We've made changes condensing the introduction and related work that hopefully answer these concerns.

Reviewer comment (Theory):
About the observation distance-based score: Since it is complicated and lacks a theoretical explanation, more intuitive explanations or visualization is needed to illustrate this formula. In many domains with image observations, partial observations or complicated robotic tasks, the Euclidean distance in the observation space may not work.

Author response:
We agree the presentation does not motivate the metric sufficiently. This metric is drawn from prior work (LSD eqn. 3, CSD eqn. 15) where such Euclidean and Mahalanobis distances were shown to have good properties in state-based domains with complex embodiment like robot manipulation (e.g. Fetch and Kitchen). We have changed the text to make this more clear. We agree that non-state-based domains may present an issue, which we discuss in Limitations, and ultimately believe using a visual latent space is the correct approach. Given prior work (DIAYN, LSD, CSD, LfP, SFP, SPiRL) began in state-based domains and sparse-reward is already difficult in state-based domains, we thought it best to tackle one problem at a time.

Reviewer comment (Experiments):
Soundness of experiments — Lack a state-based baseline in Table 1 experiments, like SPiRL. SPiRL reaches a return of 2.9 [1] in Kitchen, which outperforms the proposed method a lot. It may also perform well in other environments.

Author response:
It appears our presentation was not sufficiently clear to motivate these choices in the original text. We've made large changes to Section 4.1 and 4.3 to motivate our choices, and move experiments comparing to conditioned skills into the main text.

The reason that we see the correct baseline as SSP and not SPiRL is that we extract skills without any information on when to use those skills, then learn an RL policy on top of those skills. SSP does the exact same with a VAE. SPiRL extracts both skills **and** the knowledge of when to use those skills in the environment, which leads to faster and more stable learning when the final task aligns well with the demonstration data, but handicaps learning in new environments. If we wanted to compare with observation-conditioned methods, the correct approach would be to design a prior that looks at the particular observations when our subwords are used, and biases exploration with that prior like SPiRL. As we did not implement such a method, it seems correct to compare directly against SSP. Still, we take this point into account and have added experiments in Section 4.3 with comparison to conditioned skills and their generalization behavior.

Reviewer comment (Evaluation):
Soundness of experiments — Training curves of returns are usually required for RL experiments, which depict both performance and sample efficiency. However, this paper only presents the return at the end of training in Table 1, which can be doubtful due to the instability of RL algorithms. The data in Table 1 have large variance and are therefore unreliable.

Author response:
We are happy to include training curves. We initially thought it unwise given all 0s that are reported in tables correspond to runs that have 0 reward the entire run, thus most of the content would be flat lines. Prior work OPAL also presents tables, so for space we thought it wouldn't be an issue. We agree that there is inherent instability in RL, which is exacerbated by sparse-reward settings. To give more information as to the actual behavior of methods regardless of reward, we include exploration in Figure 4 which supports our method. In Figure 5 we make the point that, even when reward is low, exploration is still very good, thus the issue with instability has more to do with optimization on the RL side. We have explored many existing tricks to fix this (e.g. Prioritized Replay, REDQ, RND, DDQN, extra entropy regularization, hyperparameter tuning, etc.) but were not able to achieve completely stable results. To further this analysis we include visualizations in Appendix J of a seed of one of our runs whose Q-function collapses, that achieves high reward in the middle of training and 0 at the end.

Reviewer comment (Experiments):
Lack the discussion of disadvantages of state-free skills: The paper emphasizes the generalization ability of state-free skills, but ignores its disadvantages. State-free methods are inefficient on large datasets with diverse behaviors, since a large skill space is required to model all the behaviors. In this case, state-conditioned methods like SPiRL can model possible behaviors based on the state, thus providing a compact skill space. The paper also lacks such study on the influence of data diversity.

Author response:
We agree that our presentation was likely too imbalanced. We've now made large changes to Section 4.3, including extra experiments, which discuss this in a more balanced fashion.

Reviewer comment (Experiments):
In action space clustering, the number of clusters is very small ($2\times d_{act}$ is less than 20 in AntMaze and Kitchen). Is this because the agent behavior in the dataset is not diverse? If the dataset is large or collected by various stochastic policies, a small number of clusters can make the discretized actions unable to complete the task. In this case, do we need a number of tokens proportional to $2^{d_{act}}$?

Author response:
We've now included experiments in Appendix I looking at diversity of demonstrations in for Hopper, where D4RL provides a clear axis of quality. We also provide ablations in Appendix H for varying discretization. A large number of clusters is preferred, but it is still possible with a small number. If we believed every possible articulation was necessary to model, then an exponential number of tokens makes sense, but this may not be the case in many tasks.

Reviewer comment (Presentation):
Algorithm 1 could be presented in a much clearer way by adding comments for each of the steps and what each of the variables means.

Author response:
We agree that the presentation is much too condensed, and have added comments for important steps in the revised version!

Reviewer comment (Novelty):
I am unclear on why the mean vector for each subword is calculated in this way. Is there prior work that does it this particular way, or is it a design decision? Maybe this can be elaborated in the paper.

Author response:
It is true that the presentation is quite terse. For this choice we are inspired by prior work LSD and CSD (equation 15). We have added description to the corresponding section that hopefully better motivates.

Reviewer comment (Experiments):
I think some important ablations should have been included in the experiments: Clustering-only: how far can you go with pure discretisation of the action space, without learning skills?

Author response:
It appears our communication was not as clear as it could have been. The clustering-only ablation is already in the paper and corresponds to the SAC-discrete column in Table 1, where clustered actions are used as discrete actions. We can see the poor exploration behavior in Figure 4a. We've added small changes to the text to clarify this.

Reviewer comment (Experiments):
I think some important ablations should have been included in the experiments: Alternate tokenisation strategy: if instead of having a preference for longer skills/subwords, what would the performance look like if the size of each skill was fixed? ($N_{max}$ seems to have been varied across environments, per the appendix, so this would also be an interesting ablation).

Author response:
We varied hyperparameters across environments in order to get skills of length close to 10, to be comparable to baselines. Our goal was to test the viability of the simplest possible tokenization strategy available, hence why we began with BPE, a quite standard algorithm. We also discuss some shortcomings of grammar-learning methods like Sequitur in the end of the related work. Are you suggesting a version of BPE with an additional cutoff? It is not trivial to tightly control the length of skills BPE discovers.

Reviewer comment (Theory):
The paper mentions that for frequency-based merging, the longest subword is selected and all its constituent subwords are pruned. My understanding is that this biases the approach in favour of longer subwords/skills. Could you confirm?

Author response:
Yes this is true. For more context, the maximum length of subwords ultimately is decided by the maximum number of merges allowed, which we set such that our subwords are on average the same length as the baseline SSP (10 steps). If we allowed more merges, the subwords would be even longer. Alternatively one could redesign BPE for this setting, but we wanted to start with the simplest choice.

Reviewer comment (Experiments):
Why did the kitchen dataset consist of expert demonstrations, when the other datasets did not?

Author response:
Thanks for the question, as in our answers above we used a standard dataset D4RL, whose collection we do not have control over. We thought this would be a better choice than bespoke datasets.

Reviewer comment (Evaluation):
Did you perform any analysis/have any insights on the frequency of usage of each skill for a trained policy?

Author response:
With regards to frequency of skill usage we do not have this information, but would be happy to include such an analysis.