Paper: Learning Abstract World Models for Value-preserving Planning with Options (ICLR-2024)

Reviewer comment (Experiments):
The paper lacks comparisons to many more RL algorithms, especially hierarchical ones like HIRO (https://arxiv.org/abs/1805.08296), HVF (https://arxiv.org/abs/1909.05829), and Director (https://arxiv.org/abs/2206.04114). Since the authors are proposing to learn a state encoder $\phi$, the key question that must be answered is whether this $\phi$ outperforms the encoders learned by other state-of-the-art hierarchical RL algorithms.

Author response:
- **Unconvincing experiments**: We are currently working to add MBRL baselines to our experiments. However, we think it's important to notice that HIRO and Director do *skill discovery*. We assume that skills are given and we leave for future work to use our framework in a loop of skill discovery.

Reviewer comment (Theory):
In Section 3.2, the authors claim that when B = \bar{B}, "then simulating a trajectory in the abstract model is the same as in the ground model". This is not necessarily true, because rewards also need to match between the two trajectories, and $B_t$ says nothing about rewards, only dynamics. The further claim that planning in the abstract model yields the same value of an abstract state z as in the ground MDP is also not generally true; it relies on the additional assumption $E_{s \sim G(\cdot \mid z)}[R(s, o)] = \bar{R}(z, o)$, and under this assumption plus Definition 3.6 we end up back at the bisimulation conditions, making the contributions unclear.

Author response:
- **Issues in technical details**:
We understand the concern on the relationship between dynamics-preserving abstractions and bisimulation (and similar), and we have updated our writing to help with this. 

- However, we argue that the purpose of an abstract model for a set of skills is to be able to solve many new tasks with those skills by planning. Hence, the ground reward *is not meant to represent a task* but to represent the cost or reward associated with executing skills (e.g. energy cost to execute a particular motor skill). 
Therefore, we are not interested in preserving Markovinity with respect to the ground reward because there’s a limit to how much we can abstract away then. For instance, in the Antmaze experiment, if the ground reward is dependent on the joints' states then we wouldn't be able to abstract beyond the expert features. Therefore, we estimate the average cost of executing an option when it start at $s$ and ends in $s'$ (averaged over possible trajectories---see ground reward definition).
When we move to the abstract level, given an abstract state we don't know exactly in what ground state we would be, hence, the best we can do is to estimate the average ground reward under the grounding function, and in turn **we preserve the mean value with respect to the grounding function**. 

We thank you for pointing this out and we have updated our writing to reflect this.

Reviewer comment (Theory):
The authors often claim that a discrete abstract state space is bad, referencing work on symbolic abstraction learning (e.g., PDDL), but the reasoning is not clear. They state that discrete abstract states are "not applicable when planning with the available high-level actions requires a continuous state representation", but this does not make sense since options operate in the ground environment states and could therefore be defined with respect to either discrete or continuous abstract state spaces.

Author response:
- **Discrete vs. Continuous**:
Although there are, indeed, discrete problems that are important, as previous work has tackled, not all problems we might have interest in solving are naturally discrete. For instance, the navigation problems in our experiments show that there is a continuity of $(x,y)$ positions to navigate. While discretization is definitely possible, it is not necessarily the most useful representation. Moreover, we also show that our approach is a strict generalization of the strong subgoal property that allows to build discrete models (as done for PDDL in previous work).

Reviewer comment (Experiments):
The paper does not engage with the potential advantages of discrete abstractions. Planning in discrete abstraction spaces can be much easier (e.g., using symbolic planners). A fruitful research direction would be to compare the abstractions learned by symbolic approaches against those learned by continuous approaches like the authors'.

Author response:
- **Discrete vs. Continuous**:
Although there are, indeed, discrete problems that are important, as previous work has tackled, not all problems we might have interest in solving are naturally discrete. For instance, the navigation problems in our experiments show that there is a continuity of $(x,y)$ positions to navigate. While discretization is definitely possible, it is not necessarily the most useful representation. Moreover, we also show that our approach is a strict generalization of the strong subgoal property that allows to build discrete models (as done for PDDL in previous work).

Reviewer comment (Novelty):
How is the dynamics preserving abstraction defined in Def. 3.6 different from the Markovian abstractions defined in [Srivastava et al. 2016]?

Author response:
Markov Abstractions: Markov abstractions deal with finding compression functions that preserve Markovianity at the primitive action level in both transition dynamics and rewards. That is, as much compression as possible at the primitive action level and task level. When we introduce temporal abstractions, we lose information by only observing (s, o, s’) (as oppose to the entire trajectory generated by the option execution). Therefore, some details of the state become less and less predictable. We leverage that to abstract beyond the point of Markovianity (as defined in the original MDP) and we preserve *Markovinity at a higher time scale*. However, in that process we lose information about the ground state (given an abstract state we cannot know exactly in what ground state we would be in) and, hence, we cannot fully predict the ground reward function. And we don’t want to preserve the ground reward predictability! because we want to use our abstract model for new tasks that we can solve with our options.  

How is the dynamics preserving abstraction defined in Def. 3.6 different from the Markovian abstractions defined in [Srivastava et al. 2016]?  
Thank you for pointing this reference out! I believe this shares the spirit of abstraction we are proposing in this paper: we can introduce impreciseness without being incorrect. Abstracting beyond the Markovianity of the ground MDP, introduces impreciseness on what ground state will be when an option is executed in the real environment, but preserves the relevant information to execute the next option. For instance, in the antmaze experiment, the model preserves knowledge of the ant’s position in the maze but cannot predict precisely with what pose the agent will reach such position.

Reviewer comment (Writing):
What is $P(s'|s,o)$ used in the paragraph right after definition 3.1?

Author response:
**1) and 2)** We use $T(s’|s,o) = Pr(s’|s,o)$. (we have updated this in the notation to use only T). Moreover, we use the expected-length model of options. Hence, what we’re saying is that we execute the option as a black box, and observe how long it took ($\tau$) and where it ended at s’, and we model the option’s mean duration and the observed transition probabilities $Pr(s’|s, o)$ independently. Hence, the effect of the initiation sets and termination conditions are subsumed by the model of options.

Reviewer comment (Theory):
An option $o$ is defined, and then you mention $T(s'|s,o)$ to define the transition probability of taking option $o$ in $s$? $T$ earlier was defined on action space $A$. How is it applied on options without showing the relationship of $I_o$ and $\beta_o$ with $s$ and $s'$ under option policy $\pi_o$?

Author response:
**1) and 2)** We use $T(s’|s,o) = Pr(s’|s,o)$. (we have updated this in the notation to use only T). Moreover, we use the expected-length model of options. Hence, what we’re saying is that we execute the option as a black box, and observe how long it took ($\tau$) and where it ended at s’, and we model the option’s mean duration and the observed transition probabilities $Pr(s’|s, o)$ independently. Hence, the effect of the initiation sets and termination conditions are subsumed by the model of options.

Reviewer comment (Experiments):
The main focus of the paper is to show the efficiency of planning and learning when using the proposed abstract MDP. The experiments in the paper are a bit simple to showcase the benefits of the abstract model for planning. It would be stronger if the experiment was done in more complex environments with much longer-horizon tasks, such as AntMaze experiments (Hafner 2022) or robotic manipulation tasks [a].

Author response:
- **Environment choice**: It is true that we can choose more intricate domains. However, I would argue that our Antmaze domain (even though not egocentric as the one referenced) shows something interesting: the state vector is an expert feature vector that is Markov by design. However, introducing skills for navigation allow us to build a model that abstract away (automatically) even more details and let the agent concentrate in the only important features for maze navigation (the ant's position in the maze).

Reviewer comment (Experiments):
Similarly, the comparisons in Figure 5 are essentially between model-free RL (ground) and model-based RL (abstract), which does not seem fair. It might be fair to compare the proposed method with other model-based RL approaches, such as Dreamer and TD-MPC.

Author response:
- **Model-free vs. Model-based**:
It is worth pointing out that the comparison between DDQN on the ground MDP and DDQN in the abstract MDP shows that “losing” the information from a Markov state does not impede the DDQN agent finding a optimal policy (the DDQN is only ever trained with trajectories from the abstract model). This is because the retained information is the only relevant information.
We, however, understand the concern about the evaluation between model-free and model-based comparisons. *Currently, we are working on a Dreamer-like baseline that we will add to the draft asap*.

Reviewer comment (Experiments):
Some highly relevant works on temporally-extended models [a,b] are missing in the paper. Proper comparisons to these approaches are necessary.

Author response:
- **Missing references**: We have added this references to our related works sections! thanks!