Paper: Skill Machines: Temporal Logic Skill Composition in Reinforcement Learning (ICLR-2024)

Reviewer comment (Writing):
The definition of (linear) temporal logic is not provided.

Author response:
- We have added the definition for LTL in Section 2.1 page 3. We thank the reviewer for this suggestion. Due to space constraints,  we refer the reader to Camacho et al. (2019) for an expanded explanation of LTL and its interaction with RL.

Reviewer comment (Writing):
I would suggest the authors to briefly mention how to convert LTL to reward machines, e.g., through buchi automata, to make it more clear.

Author response:
- Similarly, we have added a brief description of how to convert LTL to reward machines in the caption of Figure 1. Due to space constraints,  we again refer the reader to Camacho et al. (2019) for an expanded explanation of how to convert LTL to reward machines.

Reviewer comment (Novelty):
This work only considers some simple temporal operations such as "until" and "eventually", but no continuous time interval. Extension to other temporal logic like signal temporal logic (STL) can be more interesting and useful practically -- also more challenging for sure.

Author response:
- We focus on LTL because of its relative simplicity and popularity in the literature, which is integral to not obfuscate the focus of this paper—which is about addressing the spatial and temporal curses of dimensionality by composing sub-tasks logically and temporally. Extending this work to other types of temporal logic like STL is definitely an interesting direction for future works. Since such an extension is not trivial, it is outside the scope of this work, but we believe that this work forms a good foundation for such extensions (since STL is a generalisation of LTL).

Reviewer comment (Experiments):
There are some related work to compare. For instance, I find the following two papers relevant to this paper, the authors are encouraged to compare with them.  
[1]. Luo, X. and Zavlanos, M.M., 2019, December. Transfer planning for temporal logic tasks. In 2019 IEEE 58th Conference on Decision and Control (CDC) (pp. 5306-5311). IEEE.  
[2]. León, B.G., Shanahan, M. and Belardinelli, F., Systematic Generalisation of Temporal Tasks through Deep Reinforcement Learning.

Author response:
- We thank the reviewer for the suggested works, and we are happy to include them in our related works. We will further discuss them here and hope that this clarifies why we omitted them from our baselines.


- To the best of our understanding, Luo et al. (2019) assume the usual planning/optimal-control setting, which is a very different setting to ours and all of the prior works cited in this paper. Mainly, they assume that the environment dynamics are known and amenable to planning. Here, they are more interested in how to efficiently obtain and transfer plans from previous LTL tasks to new ones. Hence, they also do not compare against reward machines but instead compare against a prior work on temporal logic optimal-control. Note that in our work and prior works, the environment dynamics are not known (hence why RL is used), hence we are unsure how the suggested baseline can be applied in this setting (since planning at the level of dynamics is not possible). In our work and prior works, planning (if used) only happens at the level of the reward machines.

- We believe that CRM is representative of León et al. (2019) in our Figure 3 experiments. Since our work is the first to attempt and achieve both spatial and temporal skill composition in temporal logic tasks provably (to the best of our knowledge), the aim of this experiment was to demonstrate the importance and benefit of our approach compared to the other popular ones considered in prior works. To elaborate slightly:

  -  In León et al. (2019), assuming that the tasks are specified using TTL, they propose using a combined or separate task and observation module (LSTMs and CNNs) to end-to-end learn an embedding of “TTL progressions”. They then demonstrate empirically that their approach can learn given tasks and also achieve some generalisation to new ones, with varying success depending on the architectures used. However, CRMs and León et al. (2019) do not leverage spatial or temporal skill composition. Hence, they rely on function approximation (e.g neural networks) to generalise both spatially and temporally. Hence, considering the spatial and temporal curses of dimensionality described in our paper (and the arbitrarily long temporal sequence of sub-tasks present in each task), they have to learn every new task with high probability. Consequently, CRMs provide a baseline representative of the class of algorithms without spatial or temporal composition that instead leverages the compositional structure of temporal logic to learn optimal policies efficiently.


- Thus, we are uncertain what benefit a comparison with León et al. (2019) would provide which is not already achieved by comparing with CRMs, and would appreciate the reviewer's thoughts on this point. We do still agree that these are important citations and will gladly include them as such.

Reviewer comment (Writing):
In Sec 2.2, the authors say "Consider the multitask setting, where all tasks share the same state and action space, transition dynamics and discount factor, but differ in reward function. The agent is required to reach a set of desired terminal states in some goal space G ⊆ S." Does this mean all subtasks can be represented as reaching a some goal states? If yes, this should be remarked more clearly.

Author response:
- Yes. We thank the reviewer for the suggestion and have clarified this sentence in the background (Sec 2.2 Page 3).

Reviewer comment (Theory):
Although the paper presents a method for few-shot learning to create new skills, it remains unclear why learning a new skill is preferred over recomposing existing ones, raising questions about the approach's efficiency.

Author response:
**W2: [Baselines]** This is an important concern from the reviewer, and we hope our discussion here fully clarifies why we omitted the 2 suggested papers from our baselines (despite citing them).

- We believe that HRM [Icarte et al., 2022] is representative of LOF [Araki et al., 2021] and that CRM [Icarte et al., 2022] is representative of LTL2Action [Vaezipoor et al., 2021] in our Figure 3 experiments. Since our work is the first to attempt and achieve both spatial and temporal skill composition in temporal logic tasks provably (to the best of our knowledge), the aim of this experiment was to demonstrate the importance and benefit of our approach compared to the other popular ones considered in prior works. To elaborate slightly:

  - While approaches that only leverage temporal skill composition like HRMs and LOF are able to learn tasks fast, they all still require learning the sub-task options for most new tasks. They are also all hierarchically optimal, while our few-shot approach is globally optimal (Figure A5 and A6 shows how the performance of a hierarchically optimal policy can be vastly lower than that of a globally optimal policy).

  - In LOF, the authors learn options for each subgoal proposition of the reward machine, then use value iteration or Q-learning to learn a meta-policy over options to solve the **current** task. These options can then be reused in new tasks (i.e new reward machines) where the same sub-goal propositions appear. As described in the introduction and observed by Liu et al. (2022), in all these works (similarly to HRMs), the options learned from a previous task can not be transferred satisfactorily to some new tasks (also see the example on page 4). For example in the office world, if in a previous task the agent learns an option for “getting coffee”, it can reuse it to “get coffee then deliver it to the office”, but it cannot reuse it in a new task where it needs to “get coffee without breaking decorations”. In contrast, Skill Machines (our work) are able to bypass this issue by learning skill primitives that can then be composed zero-shot to get coffee with or without constraints. Thus, HRMs and LOF omit spatial compositionality and HRMs provide an appropriate baseline which is representative of the limitations of all such algorithms without spatial compositionally that instead leverages temporal compositionally to learn hierarchically-optimal policies. 

  - In LTL2Action, assuming that the tasks are specified using LTL, they propose using a separate LTL module (a graph neural network) to end-to-end learn an embedding of “LTL progressions”. They then demonstrate empirically that their approach converges to more optimal policies than a myopic baseline and also achieves comparable or better generalisation. However, CRMs and LTL2Action do not leverage spatial or temporal skill composition. Hence, they rely on function approximation (e.g neural networks) to generalise both spatially and temporally. Hence, considering the spatial and temporal curses of dimensionality described in our paper (and the arbitrarily long temporal sequence of sub-tasks present in each task), they have to learn every new task with high probability. Consequently, and similarly to the relationship between HRMs and LOF, CRMs provide a baseline representative of the class of algorithms without spatial or temporal composition that instead leverages the compositional structure of temporal logic to learn optimal policies efficiently.

- Thus, we are uncertain what benefit a comparison with LOF or LTL2Action would provide which is not already achieved by comparing with HRMs and CRMs, and we would appreciate the reviewer's thoughts on this point. We do still agree that these are relevant citations, reflected by the fact that we did cite them in the original version.

Reviewer comment (Theory):
It does not seem a learned new skill in few-shot learning is consistent with Definition 3.1.

Author response:
**Q6:**
- In the few-shot learning case, the new skill learned is not a skill primitive, but just a regular action-value function for the specific task. We have clarified this point on page 6 (the highlighted text).

Reviewer comment (Experiments):
One weakness of the paper lies in its relatively complex non-end-to-end training process. More precisely, it necessitates reinforcement learning training for acquiring skill primitives and additional value iteration for the selection of appropriate spatial skills.

Author response:
- Please note that the use of RL to acquire skills is standard in the literature. Hence, while the need for RL to learn the skill primitives is indeed a weakness of this work, it is also a weakness of all prior works that do not assume known environment dynamics (e.g [Vaezipoor et al., 2021; Jothimurugan et al., 2021; Liu et al., 2022; Icarte et al. 2022]). 

- However, investigating approaches for acquiring skill primitives without RL is still an interesting direction for future works. For example, by using planning when the environment dynamics are known or behaviour cloning/imitation learning/offline RL when appropriate trajectories datasets are available. 

- Please note that the planning (value iteration) done in this work is only over the reward machines (it does not include the environment dynamics), and hence uses no additional environment samples. Value iteration is also a standard algorithm in RL [Sutton & Barto, 2009].

Reviewer comment (Evaluation):
Another potential limitation is its applicability, which may be more suited to navigation scenarios. It requires different linear temporal logic (LTL) task specifications tailored to specific application scenarios, potentially leading to challenges in generalization across various scenarios.

Author response:
- We thought the reviewer's concern here was about the applicability of LTL (or temporal logic specified tasks in general), and that it may be more suited to navigation scenarios. Our citations were to address that concern. We apologise if we misunderstood the reviewer's concern here.

- Please note how the semantic nature of tasks such as navigation, arm reach, or object manipulation is not of relevance in this work and prior works. The agent does not know this semantic nature, and there is no assumption in this work and prior works that make use of that semantic nature. The agent only assumes given sensors to detect when propositions are achieved (i.e a labeling function). We have treated environment observations in our theory as coming from an arbitrary environment. Hence, the types of tasks we considered in our experiments were specifically chosen to demonstrate the generalisation of our approach across various representative practical scenarios:
  - Across different LTL specifications. For example, the LTL tasks in Table 1 for the office grid world, Table 2 for the moving targets domain, and Table A4 for safety gym.
  - Across different types of observation spaces. For example, tabular (office grid world), RGB pixel observations (moving targets), and continuous vectors (safety gym).
  - Across different types of action spaces: discrete actions (office grid world, moving targets) and continuous actions (safety gym)
- Consider for example a robot arm manipulation scenario with 3 propositions (e.g each for putting an object in each of 3 bins), unknown continuous dynamics, vector observations, and vector actions. The safety gym tasks are representative of this scenario.
- Finally, while we no longer have enough time left in the rebuttal period to include a manipulation task experiment, we hope this explanation is sufficient for the reviewer.  

**Generalisation**
- We thought the reviewer's concern here was: because of the *applicability of LTL* concern above, there may be challenges in generalization across various scenarios (in the sense that our approach, which is focussed on temporal logic specified tasks, may not apply to various scenarios). Our citations were to address that concern. We apologise again if we misunderstood the reviewer's concern here.
- Hence, can the reviewer please clarify what their *generalisation* concern is? We are dedicated to making our work as clear as possible.

We thank the reviewer again for their engagement and for giving us the opportunity to elaborate more on our choice of tasks as we did take great care to use the most relevant ones for this paper. We look forward to the reviewer's thoughts on this point and the generalisation one.

Reviewer comment (Writing):
The descriptions of definitions 3.1 and 3.2 are not clear enough and lack some corresponding explanations.

Author response:
- We have added Figures A7 and Figure A8 to further improve the understanding of Definitions 3.1 and Definition 3.2. We hope these help clarify the reviewer’s concern here, and are happy to further clarify any aspect of these definitions.

Reviewer comment (Experiments):
How are different tasks specifically set in each benchmark? Why does the agent achieve zero-shot performance?

Author response:
- For each task in each benchmark, the RM is obtained by converting the LTL expression into an FSM using Duret-Lutz et al. (2016) (https://spot.lre.epita.fr/app/), and then giving a reward of 1 for accepting transitions and 0 otherwise. We then use the code base of Icarte et al. (2022) (https://github.com/RodrigoToroIcarte/reward_machines) to set the tasks for each RM.

- Our agent achieves zero-shot performance because at each FSM state, the learned skill primitives can be composed to achieve the Boolean expression obtained from value iteration (if it is achievable). We have added Figure A8 to make this process clearer.

Reviewer comment (Theory):
The policy generated by Skill Machine (SM) may be locally optimal, so Theorem 3.4 assumes global reachability and the policy is satisficing. There may be enough combinations of skills to approach this assumption, but it will limit the agent's exploration ability. And what is the approximate performance gap when certain states are unreachable?

Author response:
- We have added two experiments (Figure A5 and Figure A6 on page 18) where we evaluate the performance of our approach in different cases when the reachability assumption does not hold. Please see the highlighted text in Section A.4 for the discussion. In brief, we observe that the performance of the zero-shot agent may significantly decrease when the reachability assumption does not hold in some states (Figure A5), and may lead to complete failures in the worst-case scenario when the reachability assumption does not hold in all states (Figure A6). In both cases, we observe that the few-shot agent can still quickly improve on the zero-shot performance by learning the optimal policy.