Paper: Reconciling Spatial and Temporal Abstractions for Goal Representation (ICLR-2024)

Reviewer comment (Reproducibility):
I doubt that the presented results are practical. My main concern is that the _Approximating the reachability property_ step seems very costly. It's a pity that the authors didn't describe this essential part in more detail, leaving only the reference to other papers (are they published anyway?).

Author response:
The reachability computation has two steps.

*(A)* We approximate the k-steps reachability relation for a continuous state space with a neural network model $\mathcal{F}_k(s_t,G_d)$, called forward model (see Sec. 3.2). $\mathcal{F}_k (s_t, G_d)$ predicts the state the agent would reach in k steps starting from a state $s_t$ when applying the low level policy conditioned on the goal $G_d$ (i.e., the abstract state we want to reach). In practice, $\mathcal{F}_k$ is capable of generalizing the reachability relation to unvisited states. We train $\mathcal{F}_k$ from past explorations at the end of each episode (see Line 26 of Algorithm 1, Sec. E and Sec. D.3). 

*(B)* We use the set-based  reachability analysis algorithm Ai2 (T. Gehr et al., "AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation," SP 2018 and Liu et al., 2021) on the neural network $\mathcal{F}_k$ to compute the set of states that $G_i$ can reach applying $\mathcal{F}_k$ with $G_j$ (i.e., the set $\mathcal{R} := {s’ \mid \mathcal{F}_k(s, G_j), s \in G_i}$). Such an algorithm works on continuous states via efficient **set-based operation** (i.e., all the computations 
are done on sets and not on single states). We then check if $R \cap G_j^C = \emptyset$ to check the reachability property from Definition 1.

Reviewer comment (Theory):
How do you estimate the reachability for _all states_, provided that the state space is continuous?

Author response:
The reachability computation has two steps.

*(A)* We approximate the k-steps reachability relation for a continuous state space with a neural network model $\mathcal{F}_k(s_t,G_d)$, called forward model (see Sec. 3.2). $\mathcal{F}_k (s_t, G_d)$ predicts the state the agent would reach in k steps starting from a state $s_t$ when applying the low level policy conditioned on the goal $G_d$ (i.e., the abstract state we want to reach). In practice, $\mathcal{F}_k$ is capable of generalizing the reachability relation to unvisited states. We train $\mathcal{F}_k$ from past explorations at the end of each episode (see Line 26 of Algorithm 1, Sec. E and Sec. D.3). 

*(B)* We use the set-based  reachability analysis algorithm Ai2 (T. Gehr et al., "AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation," SP 2018 and Liu et al., 2021) on the neural network $\mathcal{F}_k$ to compute the set of states that $G_i$ can reach applying $\mathcal{F}_k$ with $G_j$ (i.e., the set $\mathcal{R} := {s’ \mid \mathcal{F}_k(s, G_j), s \in G_i}$). Such an algorithm works on continuous states via efficient **set-based operation** (i.e., all the computations 
are done on sets and not on single states). We then check if $R \cap G_j^C = \emptyset$ to check the reachability property from Definition 1.

Reviewer comment (Experiments):
I think that the reachability approximation model should be analyzed as well.

Author response:
We agree, our paper misses to explain the background on reachability analysis and how we instantiate it in our context. 

We will add a paragraph at the end of Section 3.1 explaining how we approximate the reachable sets using abstract interpretation with Ai2, which computes an over-approximation of the network’s output layer-by-layer using operations on an abstract domain. 

We remark that our paper *does not provide* a novel contribution for reachability analysis of neural networks, but instead uses off-the-shelf techniques. In fact, neural network reachability analysis is a known problem with multiple algorithms and tools available (e.g., see the (Verification of Neural Network competition)[https://sites.google.com/view/vnn2023] organized during CAV and AAAI).

Yes! Your understanding about over-approximating the neural network outputs is correct.

In the algorithm, we try to train a precise $\mathcal{F}_k$: we use $\mathcal{F}_k$ in the reachability analysis only when it accurately approximates the data we observed so far (see last answer from (our second response)[https://openreview.net/forum?id=odY3PkI5VB&noteId=uoi64FiwJ0]).

Yes, there are tradeoffs between the size of the $\mathcal{F}_k$ model (see Appendix D.3), its precision, and the overhead of running reachability analysis. The architecture of $\mathcal{F}_k$ is a hyperparameter that we decided via hyper parametrization on the Ant Maze environment.

Reviewer comment (Evaluation):
The practicality issues could be addressed by adding a wall-time comparison between the presented methods.

Author response:
The computational cost of the reachability analysis is negligible. We are measuring execution time on a NVIDIA RTX A5500 GPU powered computer to compare the time costs of the different environments. We are continuing to run measures of these execution times and will report in a few days the time costs over a whole run of each algorithm.
Over 3M steps, the refinement operation (which includes the reachability analysis) takes on average 0.361s (variance 0.072, maximum 0.80s) for an oracle of 2 variables, 0.423s for 3 variables (variance 0.026, maximum 0.944s), 0.429s for 5 variables (variance 0.053, maximum 1.637).

Compared to other reachability based methods (HRAC) our reachability approximation is much less costly and scalable to more variables. Running HRAC over 3M steps, we observed that the reachability relation learning step takes an increasing duration. Our measures average for the last 100 calls at 49.27s (variance:15.468 and maximum:52.206s) for 2 variables, 85.140s (variance: 12.667 max:89.940s) for 3 vars, and to 902.373s for 5 variables (the execution time continues to grow exponentially in this setting). 

A key difference is that the reachability relations are approximated over sets of states in STAR and not single states as in HRAC. HRAC stores all pairs of visited states in a transition matrix which is then used to repeatedly train a neural network predicting if a state reaches another. This creates a large memory constraint. The objective function of this model has to minimize the error of predicted relations compared to the transition matrix which becomes exponentially costly since it searches all the paths encoded in the matrix, which grows with more visited states (the size of the matrix increases, and the matrix is less sparse with added transition relations). On the other hand training the forward model in STAR is straightforward and uses an MSE loss to predict reached states from the replay buffer, and neural network reachability analysis extends the computation of the forward model to predict reached sets of states.

Reviewer comment (Theory):
I would like to read here a discussion of the importance of Theorem 2, including: why does it make your method sound (recalling that you prove there exist _some refinements_, not necessarily the one you actually do in the algorithm, right?) and why it is not trivial to construct such an abstraction (e.g. by partitioning the space into any $\varepsilon$-diameter sets).

Author response:
Thank you, we see the ambiguity in the use of “there exists” in Theorem 2. A more precise statement of Theorem 2 is:
“Given an initial set-based abstraction $\mathcal{N}$ and assuming $\mathcal{N}(g_m)$ satisfies the Conditions (1), (2), and (3) of Definition 3, we compute a reachability-aware abstraction after a finite number of **reachability-aware refinements**.”
Theorem 2 proves that applying the reachability-aware refinement **we use in the algorithm** for a finite number of times we compute a reachability-aware abstraction.

Your observation about the existence of a trivial refinement is correct: that refinement would halve the grid in each dimension so that, for example, in 2 dimensions we’ll have 1 cell initially, then 4 cells, then 16 cells, … (a uniform grid size that is reduced by epsilon may also work). However, the trivial refinement *is not what we use in the algorithm* and it’s not useful in practice, since it does not respect the dynamic of the lower-level policies!

We clarify a misunderstanding on Theorem 1, Theorem 2, and Lemma 1. Theorem 1 is important on its own and proves that, if we have a reachability-aware abstraction,  we learn a sub-optimal policy. So Theorem 1 **is not used to prove** Theorem 2. 
Then, Theorem 2 proves that the refinement computes the hypothesis of Theorem 1.
We will improve Theorem 2 wording as above, integrate this discussion in Section 4, move Lemma 1 in the Appendix and explain Theorem 2’s proof informally.

Reviewer comment (Theory):
How do you estimate the reachability for _all states_, provided that the state space is continuous?

Author response:
The reachability computation has two steps.

*(A)* We approximate the k-steps reachability relation for a continuous state space with a neural network model $\mathcal{F}_k(s_t,G_d)$, called forward model (see Sec. 3.2). $\mathcal{F}_k (s_t, G_d)$ predicts the state the agent would reach in k steps starting from a state $s_t$ when applying the low level policy conditioned on the goal $G_d$ (i.e., the abstract state we want to reach). In practice, $\mathcal{F}_k$ is capable of generalizing the reachability relation to unvisited states. We train $\mathcal{F}_k$ from past explorations at the end of each episode (see Line 26 of Algorithm 1, Sec. E and Sec. D.3). 

*(B)* We use the set-based  reachability analysis algorithm Ai2 (T. Gehr et al., "AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation," SP 2018 and Liu et al., 2021) on the neural network $\mathcal{F}_k$ to compute the set of states that $G_i$ can reach applying $\mathcal{F}_k$ with $G_j$ (i.e., the set $\mathcal{R} := {s’ \mid \mathcal{F}_k(s, G_j), s \in G_i}$). Such an algorithm works on continuous states via efficient **set-based operation** (i.e., all the computations 
are done on sets and not on single states). We then check if $R \cap G_j^C = \emptyset$ to check the reachability property from Definition 1.

Reviewer comment (Evaluation):
What is the computational cost of refining the abstractions?

Author response:
The computational cost of the reachability analysis is negligible. We are measuring execution time on a NVIDIA RTX A5500 GPU powered computer to compare the time costs of the different environments. We are continuing to run measures of these execution times and will report in a few days the time costs over a whole run of each algorithm.
Over 3M steps, the refinement operation (which includes the reachability analysis) takes on average 0.361s (variance 0.072, maximum 0.80s) for an oracle of 2 variables, 0.423s for 3 variables (variance 0.026, maximum 0.944s), 0.429s for 5 variables (variance 0.053, maximum 1.637).

Compared to other reachability based methods (HRAC) our reachability approximation is much less costly and scalable to more variables. Running HRAC over 3M steps, we observed that the reachability relation learning step takes an increasing duration. Our measures average for the last 100 calls at 49.27s (variance:15.468 and maximum:52.206s) for 2 variables, 85.140s (variance: 12.667 max:89.940s) for 3 vars, and to 902.373s for 5 variables (the execution time continues to grow exponentially in this setting). 

A key difference is that the reachability relations are approximated over sets of states in STAR and not single states as in HRAC. HRAC stores all pairs of visited states in a transition matrix which is then used to repeatedly train a neural network predicting if a state reaches another. This creates a large memory constraint. The objective function of this model has to minimize the error of predicted relations compared to the transition matrix which becomes exponentially costly since it searches all the paths encoded in the matrix, which grows with more visited states (the size of the matrix increases, and the matrix is less sparse with added transition relations). On the other hand training the forward model in STAR is straightforward and uses an MSE loss to predict reached states from the replay buffer, and neural network reachability analysis extends the computation of the forward model to predict reached sets of states.

Reviewer comment (Reproducibility):
How are the abstractions represented? I bet the sets are not represented explicitly.

Author response:
In our implementation each abstract state is a hyperrectangle (i.e., a multi-dimensional interval) and the abstraction is a partition of the state space that we represent with a set of *disjoint* hyperrectangles. We could substitute hyperrectangles with different abstract domains to gain precision (which we did not need), as long as the abstract states are disjoint.

Reviewer comment (Novelty):
Why is the Navigator useful? Why not just leave the Manager and Controller? What are the theoretical and practical advantages of introducing the Navigator? Especially, given that in the end there are only several abstractions, as you show in the experiments.

Author response:
The 3-layer architecture with the Navigator, Manager, and Controller is key for performance and one of our main contributions.
The Navigator learns a policy, via RL and planning, that decides what is the next *abstract state* to explore from the current state to fulfill a task. Since abstract states are finite and discrete (i.e., a *spatial abstraction*), the Navigator can solve a long-horizon task. What is difficult is to have an abstraction that “respects the lower agents dynamics”, so that the low-level policies (the Manager and Controller) manage to reach a goal $G_j$ from $G_i$. This is why we learn a reachability-aware abstraction.
While having an abstraction with **fewer states** is desirable for easier planning for long-horizon tasks, the abstractions need to be detailed enough to represent the properties of the environment and the task, thus finding the right granularity is key. For example, in Ant Maze Fall the Navigator does not need a fine-grained subdivision of the states in the pit, but only the abstraction helping to cross the pit in correspondence with the block (see Figure 4 b). Similarly, in Ant Maze Cam having a fine-grained representation of the orientation angles most useful close to the camera position.

Without the Navigator, our algorithm would not perform a spatial abstraction (similarly to HIRO).
Figure 3 shows the practical advantages of our architecture: temporal abstraction (HIRO) and spatial abstraction (GARA) alone do not scale when increasing the dimensions *of the oracle*.
Theoretically, Theorem 1 proves sub-optimality when using the reachability-aware abstraction.

Reviewer comment (Theory):
Why does Theorem 2 make your method sound (recalling that you prove there exist _some refinements_, not necessarily the one you actually do in the algorithm, right?), and why it is not trivial to construct such an abstraction (e.g. by partitioning the space into any $\varepsilon$-diameter sets)?

Author response:
Thank you, we see the ambiguity in the use of “there exists” in Theorem 2. A more precise statement of Theorem 2 is:
“Given an initial set-based abstraction $\mathcal{N}$ and assuming $\mathcal{N}(g_m)$ satisfies the Conditions (1), (2), and (3) of Definition 3, we compute a reachability-aware abstraction after a finite number of **reachability-aware refinements**.”
Theorem 2 proves that applying the reachability-aware refinement **we use in the algorithm** for a finite number of times we compute a reachability-aware abstraction.

Your observation about the existence of a trivial refinement is correct: that refinement would halve the grid in each dimension so that, for example, in 2 dimensions we’ll have 1 cell initially, then 4 cells, then 16 cells, … (a uniform grid size that is reduced by epsilon may also work). However, the trivial refinement *is not what we use in the algorithm* and it’s not useful in practice, since it does not respect the dynamic of the lower-level policies!

We clarify a misunderstanding on Theorem 1, Theorem 2, and Lemma 1. Theorem 1 is important on its own and proves that, if we have a reachability-aware abstraction,  we learn a sub-optimal policy. So Theorem 1 **is not used to prove** Theorem 2. 
Then, Theorem 2 proves that the refinement computes the hypothesis of Theorem 1.
We will improve Theorem 2 wording as above, integrate this discussion in Section 4, move Lemma 1 in the Appendix and explain Theorem 2’s proof informally.

Reviewer comment (Experiments):
It's a pity that by introducing the Manager you lose the abstraction of the Navigator. Can you instead of sampling a single subgoal, sample a few subgoals and somehow aggregate the low-level policy predictions to reach any of them?

Author response:
The Manager does not lose the abstraction but instead samples *a sequence of subgoals* that eventually reaches a target abstract state, acting as an intelligent tutoring system setting realistic goals for the controller (the Manager’s reward measures the success in reaching the abstract goal, see next answer).
If we understand correctly, the question asks if the Manager can sample multiple subgoals (i.e., a breadth-first exploration). We think this strategy, differently from our implementation, will explore “closer” states first but will struggle to learn to reach a different abstract state.

Reviewer comment (Theory):
Why are the proposed rewards sound? Without any justification, they look quite arbitrary.

Author response:
We designed the rewards following the common intuition from goal-conditioned HRL algorithms: the highest-level agent in the hierarchy is rewarded when it accomplishes the task goal. Each lower-level agent is rewarded for achieving the subgoal set by its higher-level agent.

The Navigator’s reward measures the negative distance from the task goal: the higher the reward, the closer is the agent to the task goal.
The Manager’s reward measures the negative distance of the agent from the goal set by the Navigator. However, the goal in this case is a set of states (and not a single state). Thus we need to compute the distance between a point and a set. In practice, instead of computing the exact distance to a set which can be costly, we experimented with different distances (e.g., the minimum distance from a point in the set, the distance from the center of the set). We converged, after some experiments, on the distance from the center of the set. 
The Controller’s reward measures the negative distance from the subgoal state picked by the Manager, in line with the lower-level agent’s reward in HIRO, HRAC, ….

Reviewer comment (Evaluation):
Are there any implicit properties of the abstractions that you learn? In the experiments, they seem to be always box-shaped and disjoint, is it always the case?

Author response:
We try to learn a reachability-aware abstraction (Definition 3).

Reviewer comment (Theory):
Is the splitting operation the only way to refine the abstraction? Can you also incorporate the join operation if the abstractions are too fine? Is being too fine a problem in any case? Is $\mathcal N=(x\mapsto \\\{x\\\})$ always a good abstraction?

Author response:
Currently, the splitting operation is the only operation on the abstract space. We are considering join operation for future works.
Currently, we avoid too fine abstractions through the hyperparameter “Minimal volume ratio of a split”, as mentioned in Table 3 of Annex D4. Moreover, in practice, the goal of our hierarchical representation is to drastically reduce the size of the space (from a continuous thus uncountable state space, to a finite abstraction space) to orient better exploration. Thus, with respect to this idea, a “too fine” abstraction would not be good.

Reviewer comment (Theory):
How do you handle the initial stage, when hardly any subgoal is reachable (because the agent can barely move)? Is there a danger that it will shatter the goal space arbitrarily before becoming capable of acting?

Author response:
You are correct, we cannot refine the abstraction using the forward model in the initial stages and, in general, before the low-level policies (Manager and Controller) are “stable”.

In fact, we refine the abstraction *only* when the forward model approximates the reachability relation induced by the low-level policies “well enough”, and continue with the low-level policies training otherwise. In practice, we test if the forward model predicts well the reachability relation of the latest episodes (i.e., the Mean Squared Error on the prediction of the latest 10 episodes is lower than a threshold). When the test succeeds, the forward model approximates the reachability relation obtained executing the low-level policies. So, this test avoids arbitrary splittings of the goal space.
We briefly explained this mechanism in Appendix D.4, but we will provide the above explanation in Section 3.2.

Reviewer comment (Experiments):
The results have been plotted only for 5 runs. Usually, this is not enough number of runs. Especially, in AntMaze and AntMazeCam where the results of HIRO and HRAC are close to STAR.

Author response:
> The results have been plotted only for 5 runs … not enough…

We agree the confidence in our results could increase with more runs.

We’re running more experiments and plan to obtain about 8 runs before the discussion ends for AntMaze (STAR, HIRO, HRAC algorithms) and AntMazeCam (STAR, HIRO algorithms) and then update the pdf accordingly (runs are computationally expensive *for all the algorithms*).

We will collect up to 10 runs for all the environments and algorithms before the paper acceptance deadline (both the HIRO and LESSONS algorithms provide an evaluation with 10 runs).

Reviewer comment (Evaluation):
It is not clear what a "Complex" continuous control task means. How to measure the complexity? Is it only related to scaling up the dimensionality, or is it also affected by distribution changes or even complications in the environment?

Author response:
> … what a "Complex" continuous control task means. How to measure the complexity? … affected by distribution changes …?

Indeed, task complexity is an ill-defined notion and no consensus has been reached on a single measure of task complexity, as there are many orthogonals measures to be considered. In our case, the complexity is linked to (1) scaling up the dimensionality of the state and action spaces and, in particular, the dimension of the oracle state space, as we show that we scale to 5 dimensions, compared to 2 in the SOTA; (2) the sparse reward function which translates the idea of a long-horizon task. We do not explicitly consider distribution changes.

Reviewer comment (Experiments):
It seems important to have a deterministic environment. The paper mentions that STAR is able to perform online. I assume it might be able to adjust to the stochasticity of the environment with some tricks.

Author response:
> It seems important to have a deterministic environment. … adjust to the stochasticity … with some tricks.

STAR does not have a specific mechanism to cope with stochasticity. *We share* the deterministic environment assumption with HIRO and LESSONS. Only HRAC showed to be robust to small perturbations in the environment, which is a limited form of stochasticity.

We hypothesize STAR’s abstraction to be also robust to small perturbations, since they would not drastically change the reachability relation among goals. However, we think STAR’s abstraction and reachability analysis should be extended (e.g., to quantify the environment uncertainty) to handle stochastic environments.

Reviewer comment (Theory):
I suspect this approach might suffer from high variance during learning due to the high non-stationarity between multiple layers of hierarchical agents. However, I believe there exist some tricks to alleviate the problem. For example, define the reachability for a backward model, so that it starts from the Goal set that the task goal $g*$ belongs to, and find all groups of states that FROM them, $g*$ group is reachable. This way, we might not need to go over all G_0 to G_n subset of abstract goal sets visited in an episode, but just visit the ones that lead to the G* set.

Author response:
> 3. How do you initialize $\mathcal{N}$ and a state set $\mathcal{G}$ …  to solve the task? … measurement … or is it randomly chosen based on $\pi_{\text{Nav}}$?
> … how do you deal with the high variance during training? 

*Abstract state initialization*: we have an initialization step where we (a) perform some random explorations (still without an abstract goal); (b) train a forward model (see [first answer to Reviewer awpo here](https://openreview.net/forum?id=odY3PkI5VB&noteId=i1r1FtB3Ue)) to approximate the reachability relation of the explored data; and (c) once the forward model is precise (see the second part of this answer) we obtain an initial abstraction separating the reachable and unreachable states (in the data). This resulting abstraction is still coarse (e.g., partitions the state once space over the x and y dimensions) and is mainly useful to identify an abstract state enclosing a neighborhood of the initial environment state.

*High-variance during training*: we mitigate this problem, which may cause non-stationarity of policies at different levels, both when refining the abstraction and in the Manager.

We refine the abstraction at the end of every episode **only if** the forward model approximates the reachability relation induced by the low-level policies. In practice, we test if the forward model predicts “well” the reachability relation of the latest episodes (testing if the Mean Squared Error on the prediction of the latest 10 episodes is lower than a threshold, see Section D.4). We hypothesize that, when the above test succeeds, the low-level policies are *stable* and the forward model captures the reachability relation. This heuristics helps in coping with the non-stationarity issue.

We further thank you for your suggestion about the refinement strategy (i.e., the “trick” to refine in a backward direction from the goal abstract state): we agree it may help in coping with non-stationarity and it is worth exploring in the future.

We further use off-policy correction (Nachum et al., 2018) to deal with non-stationarity in the Manager agent.

Reviewer comment (Writing):
I think there are two types of policies based on the way you are using them in this paper. One type of policy maps state to state (maybe a policy-conditioned transition probability?), and goals can be sampled from them; the other type is the common concept of policy in RL, where maps state to actions. See Q1 and Q6, please.

Author response:
> 1. In section 3.1, … Is not '~' used to show sampling from a distribution …? 
> I think there are two types of policies based on the way you are using them in this paper….

> 6. … how come that both actions and goals are sampled from policies?
  
We thank you for highlighting the imprecision with respect to the use of '~'; we improperly used it to show sampling from a policy, be it stochastic or deterministic.

We clarify the input and output of each of the goal-conditioned policies in STAR:

$\pi_{Nav}: \mathcal{S} \times \mathcal{S} \rightarrow \mathcal{G}$. This policy takes a state $s_t$, and a task goal $g^*$ and outputs an abstract goal $G_{t+k}$.

$\pi_{Man}: \mathcal{S} \times \mathcal{G} \rightarrow \mathcal{S}$. This policy takes a state $s_t$, and an abstract goal $G_{t+k}$ and outputs a subgoal $g_{t+l}$.

$\pi_{Cont}: \mathcal{S} \times \mathcal{S} \rightarrow \mathcal{A}$. This policy takes a state $s_t$, and a subgoal $g_{t+l}$ and outputs an action $a_t$.

Similarly we’d like to clarify the goal-conditioned policies used in the theoretical sections:

$\pi^*_{high}: \mathcal{S} \times \mathcal{S} \rightarrow \mathcal{S}$. This policy takes a state $s_t$, and a task goal $g^*$ and outputs a concrete goal (state) $g_{t+k}$.

$\pi^*_{low}: \mathcal{S} \times \mathcal{S} \rightarrow \mathcal{A}$. This policy takes a state $s_t$, and a subgoal $g_{t+k}$ and outputs an action $a_t$.

$\pi^*_\mathcal{N, High}: \mathcal{S} \times \mathcal{S} \rightarrow \mathcal{G}$. This policy takes a state $s_t$, and a task goal $g^*$ and outputs an abstract goal $G_{t+k}$.

$\pi^*_\mathcal{N, Low}: \mathcal{S} \times \mathcal{G} \rightarrow \mathcal{A}$. This policy takes a state $s_t$, and an abstract goal $G_{t+k}$ and outputs an action $a_t$.

Such hierarchical policies that output goals of different levels of abstractions are standard in goal-conditioned HRL with a 2-level hierarchy (e.g., see HIRO, HRAC, LESSON).

> 2. How is the composition of $\pi_{\text{Man}}$ and $\pi_{\text{Cont}}$ to generate  $\pi_{\text{low}}$? 

Focusing on the composition of $\pi_{\text{Cont}}$ and $\pi_{\text{Man}}$ to generate a hierarchical policy $\pi_{\text{low}}$, we can write:

$\pi_{\text{low}}(s_t, G_{k*[ t/k ])}) := \pi_{\text{Cont}}(s_t,\pi_{\text{Man}}(s_{l*[ t/l ]},G_{k*[ t/k ]}))$

where $[n]$ computes the floor of the number $n$.

Reviewer comment (Theory):
All in all, I have doubts about the soundness of the conclusions and proofs in the paper. I would be happy to change my rating if I learn more about my questions. I see the value of the core idea of this work.

Author response:
> 5. Are all the proofs … novel … ?

*Yes, all the proofs are novel*: GARA (Zadem et al., 2023) does not have proofs and the proofs from LESSONS (Li 2021) are specific to learning goal representations using “slow features”.

Reviewer comment (Writing):
The goal space {\mathcal G} and the state space S need to be more clearly stated. By definition, {\mathcal G} is a partition of S, initially a coarse one. In B.1. of the Appendix, it is written that "both the state and action spaces of the Navigator correspond to {\mathcal G}".

Author response:
$\mathcal{G}$ is *a partition* of the environment’s space $S$:  that is, $\mathcal{G}$ is a set of states $\{G_1, \ldots, G_n\}$, where each $G \in \mathcal{G}$, $G \subseteq S$, all the subsets are disjoint (for any $i \neq j$,  $G_i \cap G_j = \emptyset$ and the union of the subsets cover $S$, i.e., $\bigcap_{G \in \mathcal{G}} = S$).

The Navigator policy $\pi_{Nav}$ takes as input the current environment state $s_t \in S$ and a task goal state $g^* \in S$ (this is the final environment’s state the agent should reach), and outputs a goal state $G \in \mathcal{G}$. So, the action space of the Navigator is also $\mathcal{G}$.

In practice,  we initialize the partition $\mathcal{G}$ as explained in the last answer in this [message](https://openreview.net/forum?id=odY3PkI5VB&noteId=uoi64FiwJ0) to reviewer ep9K.

Reviewer comment (Writing):
It is unclear what are initially {\mathcal G} and G_t. Namely, if {\mathcal G} initially contains the whole set S and not just some subset of S, then it is clear that the next partition is chosen from {\mathcal G} or from the current G_t; otherwise, some clarification on the initial {\mathcal G} might be needed, also for actions.

Author response:
$\mathcal{G}$ is *a partition* of the environment’s space $S$:  that is, $\mathcal{G}$ is a set of states $\{G_1, \ldots, G_n\}$, where each $G \in \mathcal{G}$, $G \subseteq S$, all the subsets are disjoint (for any $i \neq j$,  $G_i \cap G_j = \emptyset$ and the union of the subsets cover $S$, i.e., $\bigcap_{G \in \mathcal{G}} = S$).

The Navigator policy $\pi_{Nav}$ takes as input the current environment state $s_t \in S$ and a task goal state $g^* \in S$ (this is the final environment’s state the agent should reach), and outputs a goal state $G \in \mathcal{G}$. So, the action space of the Navigator is also $\mathcal{G}$.

In practice,  we initialize the partition $\mathcal{G}$ as explained in the last answer in this [message](https://openreview.net/forum?id=odY3PkI5VB&noteId=uoi64FiwJ0) to reviewer ep9K.

Reviewer comment (Writing):
The action space is said to be in {\mathcal G}, and it needs to be explained if it is in {\mathcal G} regarding the history or possible actions. Thus, it would be better to clarify R_{max}.

Author response:
The Navigator policy $\pi_{Nav}$ takes as input the current environment state $s_t \in S$ and a task goal state $g^* \in S$ (this is the final environment’s state the agent should reach), and outputs a goal state $G \in \mathcal{G}$. So, the action space of the Navigator is also $\mathcal{G}$.

> clarify $R_{max}$

$R_{\text{max}}$ is the maximum distance between two states in the environment that are reachable in 1 step, and this value bounds the reward signals at each step.

Reviewer comment (Theory):
It is unclear how just assuming {\mathcal N}(g_m), with the required conditions of Definition 3, cannot happen at a splitting point of a chosen G_i that no state satisfies the reachability property. In particular, what is M in the last but one line on page 6?

Author response:
We do not understand this question completely, we’d be glad  if you could provide more context.

Here, we try to answer explaining what happens during a refinement. We have a sequence of abstract states $G_1, G_2, \ldots $ that we visited in the last episode.  We check the reachability property between every $G_i, G_{i+1}$ in the sequence. If the reachability property does not hold, then we split the abstract state, otherwise not. If all the pair of states satisfy the reachability property, then we do not split the abstraction.

Then, in Lemma 1, we show that if $\mathcal{N’}$ is a reachability-aware refinement of  $\mathcal{N}$, then it contains a new abstract state $G_i$ that “separates” points from the optimal high-level trajectory.

> In particular, what is $M$ in the last but one line on page 6? 

$M$ is the deterministic environment.

Reviewer comment (Writing):
The sentence on page 4, after the Navigator reward, "The max in this reward is computed over observed exploration data", is not entirely clear — "observed" in which sense?

Author response:
The Navigator’s reward for $s_t \in G_{t+k}$ is the maximal reward obtained by any state visited inside $G_{t+k}$ during the past $k$ steps. Since the Navigator operates every $k$ steps, multiple (environment) states can be visited inside $G_{t+k}$ before the reward of the Navigator $r_{Navigator}$ is computed.

Reviewer comment (Theory):
In Definition 2, after removing G_i from {\mathcal G}_{\mathcal N} and adding G_1' and G_2', it is unclear why it is necessary to specify that G_i is the union of the two after G_i has been removed and since only G_1' is chosen.

Author response:
This operation is motivated by our initial presentation of the abstraction as a **partition** of the state space $\mathcal{S}$.

It is true that $G’_1$ is the one that reaches $G_j$, but $G’_2$ is not discarded as a potential abstract goal that can be explored by the agents (i.e., it is still part of the partition).

Reviewer comment (Theory):
It seems that to do the exploration for checking the reachability property (page 5), the complete partition of the states is required, but how the exploration scales with the dimension of S is not discussed.

Author response:
The exploration using the abstract state space is challenging in high-dimensional environments (i.e., it is difficult to explore an abstract state from another one). This is in fact the main issue of the GARA algorithm that we address in STAR by adding a temporal abstraction mechanism.

Reviewer comment (Theory):
Appendix B.2 discusses the growing state space, but there is no discussion about the computational cost of the exploration given the state space partition.

Author response:
We have conducted a preliminary study of resources used by the critical approaches with close performances in AntMaze and AntMazeCam, and we report runtimes (measured in seconds) and reachability learning time cost (if applicable), averaged over 3 runs.

for AntMaze:
- STAR: 32287s, variance: 412, the reachability analysis takes 0.01% of the total runtime
- HRAC: 38997s, variance: 302, the reachability analysis takes 15.33% of the total runtime
- HIRO: 30189s, variance: 343

for AntMazeCam:
- STAR: 32760s, variance: 312, the reachability analysis takes 0.013% of the total runtime
- HRAC: we did not measure this, this approach however is very costly on this setting since it does not scale well to higher dimensions. Learning reachability relations between states requires an exponentially large search.
- HIRO: 30999s, variance: 274

Reviewer comment (Presentation):
Figure 4 is not so exciting; it does not clearly show that "progressively, the ant explores trajectories leading to the goal of the task." The representation of the exploration of the Ant Maze environment given in Nachum et Al. ICLR 2019 was quite appealing, and the pictures in this work could be improved.

Author response:
We see the representation from Figure 2 in (Nachum et al., ICLR 2019): we will provide a similar figure for the policy we learn after 3M timesteps. Note that Figure 2 in  (Nachum et al., ICLR 2019) does not show how the abstract goal representation evolves during training.