Paper: Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
Skipper appears to be designed primarily with gridworld environments in mind and seems relatively limited in the range of problems it is suited to solve.

Author response:
*Thank you for your comments. We would like to acknowledge that we indeed have only a single family of gridworld environments. This is not only due to their appropriate nature to demonstrate the coherence of the empirical behaviors of Skipper with the motivating theory, but also a product of limited page contents as well as computational resources. Despite that we authors could not disclose our identities, we would like to disclose the fact that **this work is purely done in an academic setting** without many computational resources. We genuinely suggest that you could take this as a first step towards a more generally applicable algorithm that can be empirically validated to work well in other environments.*

*Despite that Skipper isn't tested on other environments, there is no evidence that Skipper would have no applicability elsewhere, as it makes little assumptions on the kind of tasks it is meant to solve, and with absolutely no assumption for gridworlds or incompatibility to extend to partially observable scenarios. The checkpoint generation (vertex proposal in your language) was, with consideration, implemented to work on visual observations to take full advantage of the recent progress in conditional (image) generation, guaranteeing its wider applicability. For non-visual environments, the checkpoint generator should be implemented with appropriate architectures that could handle the corresponding representations of observations. We showcased a Skipper implementation for visual inputs due to their dominant popularity in the current RL community.*

Reviewer comment (Presentation):
The paper is missing a clear "Weaknesses" or "Limitations" section, which would help the reader understand when Skipper will be appropriate.

Author response:
*Thank you very much for your suggestions. We have added a whole section on weaknesses and limitations of this work.*

*Contrary to your comments, Skipper indeed operates on image-like spaces. The difference being that in our experiments, we respected the original BabyAI literature to use a low-resolution image as the observation (12x12). Again, we don’t think that we made additional implicit assumptions of a certain type of representation. However, we have added more clarifications regarding this point in the experiment section:*

> Skipper is implemented to be compatible with image inputs. In our experiments, the observations are low-res two-channel images (12x12x2). More details are presented in the Appendix.

Reviewer comment (Experiments):
The experiments focus somewhat narrowly on a single family of gridworld environments, raising questions about how applicable Skipper is to tasks beyond those demonstrated in the paper.

Author response:
*Thank you for your comments. We would like to acknowledge that we indeed have only a single family of gridworld environments. This is not only due to their appropriate nature to demonstrate the coherence of the empirical behaviors of Skipper with the motivating theory, but also a product of limited page contents as well as computational resources. Despite that we authors could not disclose our identities, we would like to disclose the fact that **this work is purely done in an academic setting** without many computational resources. We genuinely suggest that you could take this as a first step towards a more generally applicable algorithm that can be empirically validated to work well in other environments.*

*Despite that Skipper isn't tested on other environments, there is no evidence that Skipper would have no applicability elsewhere, as it makes little assumptions on the kind of tasks it is meant to solve, and with absolutely no assumption for gridworlds or incompatibility to extend to partially observable scenarios. The checkpoint generation (vertex proposal in your language) was, with consideration, implemented to work on visual observations to take full advantage of the recent progress in conditional (image) generation, guaranteeing its wider applicability. For non-visual environments, the checkpoint generator should be implemented with appropriate architectures that could handle the corresponding representations of observations. We showcased a Skipper implementation for visual inputs due to their dominant popularity in the current RL community.*

Reviewer comment (Experiments):
Skipper's vertex generation module consists of an attention mechanism over the observation, and in the experiments, the observations are overhead views of maps that directly correspond to the state space, making Skipper particularly well-suited for such environments but potentially over-engineered for them.

Author response:
***We would also like to present some evidence for you to kindly reconsider whether Skipper is over-engineered to work on these experiments. In fact, we were aware of the potentials of over-engineering and hence were adopting consciously the design choices to avoid such issue***: 

1. *As you pointed out, we acknowledge that the experiments are done in a fully-observable setting, where no recurrent / memory mechanisms need to be incorporated. We preferred not to have a partially-observable setting since the contributions of this work will not be further validated, nor would more useful insights be excavated; On the other hand, incorporating recurrent / memory mechanisms into the implementations will not only significantly slow down the experiments (excuse us again for our academic setting) but also truly make Skipper more sensitive to hyperparameters and potentially more overfit on these experiments. **Fundamentally, Skipper’s theory and framework is not limited to fully-observable settings as one could augment the encoder with recurrent / memory mechanisms and let the checkpoint generator work on the level of state representations (in our current experiments, working on the observations and the state representations are equivalent, due to full observability)***

2. *Contrary to what you said, **the vertex generation model doesn’t consist of an attention mechanism**. The attention (spatial abstraction) is only used for the estimators (cumulative value, distance, checkpoint achieving policy). Our vertex generation model is implemented as a simple conditional CVAE with the default hyperparameters that could work well for generic visual inputs (as well as convoluted feature maps, confirmed by recent advances on latent diffusion). In this paper, the observations are compact low-res images. We believe that, besides dealing with partial-observability, adapting Skipper to other visual domains only requires scaling up the architectures, which we had instead intentionally tuned down for acceleration.*

3. *Skipper’s architecture is simplistic and bear few changes to the pervasively used distributional DDQN. The major changes include the incorporation of the local perception field and a standard CVAE for checkpoint generation. While minor changes include shrinking down the architectures. All other hyperparameters and the training procedures are not changed. The local perception field was implemented minimalistically: a conv layer (no padding) followed by a depth-1 attention. These were intended to maximize its possible applicable scenarios and to avoid overfitting to the showcased tasks.*

Reviewer comment (Theory):
Unlike other model-based reinforcement learning approaches such as MuZero and Dreamer, which work in general observation spaces and can learn models from RGB images, Skipper seems to implicitly assume a certain type of state or observation representation, thus limiting its general applicability.

Author response:
*Thank you very much for your suggestions. We have added a whole section on weaknesses and limitations of this work.*

*Contrary to your comments, Skipper indeed operates on image-like spaces. The difference being that in our experiments, we respected the original BabyAI literature to use a low-resolution image as the observation (12x12). Again, we don’t think that we made additional implicit assumptions of a certain type of representation. However, we have added more clarifications regarding this point in the experiment section:*

> Skipper is implemented to be compatible with image inputs. In our experiments, the observations are low-res two-channel images (12x12x2). More details are presented in the Appendix.

Reviewer comment (Theory):
There is a need for clarification and explicit description of the implicit conditions and assumptions underlying Skipper’s design, as this would strengthen the paper and better define its applicability.

Author response:
*Thank you! To expand beyond the newly incorporated assumptions and limitations section:* 

*Technical: optionally a well-defined distance measure for identifying the achievement of checkpoints. For example, in our discrete state space experiments, the achievement of checkpoints can be either identified by comparing the identity between the target checkpoint and the current state, or their partial descriptions (currently used). For continuous state spaces, where a strict identity cannot be established, we either must adopt a distance measure for approximating the equivalence between the target checkpoint and the current state, or we continue to use the equivalence of the partial descriptions. We intentionally designed the partial descriptions to be in the form of bundles of binary variables, so that this comparison could be done trivially for any forms of the state space, making the requirement for the state distance measure optional.*

*Theoretical: while Skipper does not explicitly pose assumptions about environments, our motivating theory only guarantees good performance in the cases where the conditions of Theorem 1 could hold. These conditions however do not simply correspond to a simple family of tasks, e.g. sparse-reward navigation, etc. We validated Skipper’s coherence with the theory on sparse-reward navigation, which is a case where the just-mentioned conditions can be met via learning. We do not know if our current theory truly captures the bounds of the applicability of our framework. Potentially, we could find better theory in the future that could lead us comfortably out of these condition requirements. We also couldn’t discover and investigate Skipper’s performance in all the cases that correspond to the existing theory. We would gladly resume our journey for deeper understanding.*

Reviewer comment (Presentation):
Although the paper discusses Skipper’s effectiveness at length, it lacks explicit discussion on its weaknesses or limitations, and such a section would help clarify which problems or domains Skipper is best suited for.

Author response:
*Thank you very much for your suggestions. We have added a section on weaknesses and limitations accordingly (due to the page limit, it is placed as of now in the Appendix of the revised manuscript):*

> We generate future checkpoints at random by sampling the partial description space. Despite the post-processing such as pruning, the generated checkpoints do not prioritize on the predictable, important states that matter the most to form a meaningful long-term plan. We would like to continue investigating the possibilities along this line;

> The current implementation is intended for pixel input fully-observable tasks with discrete state and action spaces. Such a minimalistic form is because we wish to isolate the unwanted challenges from other factors that are not closely related to the idea of this work, as well as to make the agent as generalist as possible. On the matter of the compatibility with continuous spaces, Skipper is naturally compatible with continuous actions spaces and the only thing we will need to do is to replace the baseline agent with a compatible one such as TD3 (Fujimoto et al., 2018); on the other hand, for continuous state spaces, the identification of the achievement of a checkpoint becomes tricky. This is due to the fact that a strict identity between the current state and the target checkpoint may be ever established, we either must adopt a distance measure for approximate state equivalence, or rely on the equivalence of the partial descriptions (which is adopted in the current implementation). We intentionally designed the partial descriptions to be in the form of bundles of binary variables, so that this comparison could be done fast and trivially for any forms of the state space; for partial observability, despite that no recurrent mechanism has been incorporated in the current implementation, the framework is not incompatible. To implement that, we will need to augment the state encoder with recurrent or memory mechanisms and we need to make  the checkpoint generator directly work over the learned state representations. We acknowledge that future work is needed to verify Skipper’s performance on the popular partially-observable benchmark suites, which requires the incorporation of components to handle partial observability as well as scaling up the architectures for more expressive power;

> We do not yet know the precise boundaries of the motivating theory on proxy problems, since it only indicates performance guarantees on the condition of estimation accuracy, which in turn does not correspond trivially to a set of well-defined problems. We are eager to explore, outside the scope of sparse-reward navigation, how this approach can be used to facilitate better generalization, and at the same time, try to find more powerful theories that guide us better.”

Reviewer comment (Experiments):
Clarification is needed on the size of map or problem Skipper would be expected to handle successfully, including whether it would work for larger occupancy grids and navigation tasks.

Author response:
*For your additional questions: with the same architecture and hyperparameters, we have tested Skipper on sizes ranging from 8x8 up to 16x16 (with number of (post-pruning) checkpoints scaling linearly from 8 to 16). Among all these experiments, there are always clear advantages in terms of generalization abilities compared to the other agents. We particularly chose 12x12 because of our limited computational budget, as well as a clear presentation of our results: when the sizes of the problems are small, all methods perform relatively well, the advantages of Skipper cannot be convincingly demonstrated (overlapping confidence intervals); this is also the case in which all methods behave poorly when the size is too large. Additionally, the prohibitive computational demand (due to scaling up the architectures as well as the complexity of the planning process, i.e. quadratically with the number of checkpoints) would cost us several months more to finish the experiments.*

Reviewer comment (Writing):
In my opinion, the paper lacks an overall description of the algorithm which largely limits the understanding of the paper.

Author response:
*We appreciate your comments, and apologize if we hadn't done well enough on the previous attempt: we not only provided a detailed description of each algorithm component, but also an overall framework chart (Fig. 2). In addition, in the appendix, we provided an intuitive overall pseudocode to precisely assemble all components together (whose location we also mentioned in the manuscript. The page limit forced us to compromise on what could be presented in the manuscript and what could be kept in the Appendix). For your convenience, you can access its screenshot here: https://github.com/AnonymousAuthors21/ICLR2024-Rebuttal/blob/main/pseudocode.png*

*We'd also like to point out that Reviewers muKb, 3j7M explicitly remarked positively on the paper's clarity.* 

*In response to your concern, we have added more descriptions and references for the readers to quickly navigate to these contents. If there are remaining things that are unclear, please let us know and we’ll add further clarifications. Thank you!*

Reviewer comment (Novelty):
There is much research done on learning temporal and/or state abstractions automatically for planning and learning [1,2,3,4]. It is not clear how the proposed work is different or novel compared to these different lines of research. The related work needs a thorough justification of differences with different categories of work.

Author response:
*We apologize if we hadn’t done a well enough job in such matter. In the submission, Section “Related Works” (right AFTER experiments section), we tried our best to differentiate and justify our method from 5+ aspects and 12+ related works. Now, to improve, we have additionally incorporated your suggested papers in the appropriate locations:*

***[1]**: The paper’s main contribution is a stochasticity compatible HP framework that does not relate to temporal or spatial abstractions. We have added it to the manuscript in the Appendix where we verified Skipper’s compatibility with stochastic environmental dynamics.*

***(this reply is to be continued in the next part of the response)***

*[2,3]: The two papers’ main contribution is a special form of state abstraction (not spatial abstraction) that is dynamic state space partitioning, one through conditional abstraction trees and another a state clustering algorithm. Contrastively, our spatial abstraction (not state abstraction) focuses on establishing partial states over partial information of any selected decision timing for the agent to be able to focus on a sub-task.* 

*These two works above are less related to planning, but more to HRL. The recommended works’ state partitioning ideas are similar to that of the **planning** framework proposed in (Zhang et al., 2021), against which we have compared our method extensively in the related works section. Despite the use of the word “abstraction”, the two works here do not relate to our proposed spatial abstraction which is “dynamically focusing attention on relevant aspects of the state space”. We have added the acknowledgement of these works into the related works section for the comparison between the similarities and differences of the discussed HRL methods. We also added arguments to clarify the definition of spatial abstraction in this paper.*

 
*[4]: This paper contributed to proposing subgoals using by identifying bottlenecks in the state space, with no relevance to spatial and temporal abstractions. We have added acknowledgements in the related works section.*

***While learning abstractions is investigated in the literature, the contribution of this paper is fundamentally different from any existing work to the best of our knowledge.*** 

*For your convenience, here is a copy of the new version of the related works section:*

*https://github.com/AnonymousAuthors21/ICLR2024-Rebuttal/blob/main/related_work_updated.png*

*While we cannot acknowledge all relevant works in the manuscript, again, we try our best to discuss in detail the related representative works. **Hopefully, after incorporating your suggestions, your concerns regarding the relationship between this work and others can be addressed**. Thank you very much!*

Reviewer comment (Experiments):
The proposed approach seems applicable to grid-like environments only as also included in the experiments.

Author response:
*We acknowledge the fact that the experiments done for this work are only in the gridworld environments. In the paper, we had to choose a DP-solvable environment to demonstrate the coherence of empirical performance with our motivating theory. Due to the page limit, we could not include more experimental contents in the submission.*

*However, despite that we only tested on gridworld experiments, Skipper is not designed to specifically address gridworld problems, and we believe that **there is no reason to deny its applicability on other environments without evidence**. In fact, we are already working on its extension to ProcGen, which is the only visual benchmark that we know that could provide limited insights into the agents’ generalization abilities. The submission is intended to provide a basis for the future extension of its experimental scenarios, and we hope that its own contributions can be recognized.*

*We have also added a section dedicated to the weaknesses and limitations for clearer understanding of this work (due to the page limit, it is placed as of now in the Appendix of the revised manuscript, here is a copy):*

> We generate future checkpoints at random by sampling the partial description space. Despite the post-processing such as pruning, the generated checkpoints do not prioritize on the predictable, important states that matter the most to form a meaningful long-term plan. We would like to continue investigating the possibilities along this line;

> The current implementation is intended for pixel input fully-observable tasks with discrete state and action spaces. Such a minimalistic form is because we wish to isolate the unwanted challenges from other factors that are not closely related to the idea of this work, as well as to make the agent as generalist as possible. On the matter of the compatibility with continuous spaces, Skipper is naturally compatible with continuous actions spaces and the only thing we will need to do is to replace the baseline agent with a compatible one such as TD3 (Fujimoto et al., 2018); on the other hand, for continuous state spaces, the identification of the achievement of a checkpoint becomes tricky. This is due to the fact that a strict identity between the current state and the target checkpoint may be ever established, we either must adopt a distance measure for approximate state equivalence, or rely on the equivalence of the partial descriptions (which is adopted in the current implementation). We intentionally designed the partial descriptions to be in the form of bundles of binary variables, so that this comparison could be done fast and trivially for any forms of the state space; for partial observability, despite that no recurrent mechanism has been incorporated in the current implementation, the framework is not incompatible. To implement that, we will need to augment the state encoder with recurrent or memory mechanisms and we need to make  the checkpoint generator directly work over the learned state representations. We acknowledge that future work is needed to verify Skipper’s performance on the popular partially-observable benchmark suites, which requires the incorporation of components to handle partial observability as well as scaling up the architectures for more expressive power;

> We are far from understanding the precise boundaries of the motivating theory on proxy problems, since it only indicates performance guarantees on the condition of estimation accuracy, which in turn does not correspond trivially to a set of well-defined problems. We are eager to explore, outside the scope of sparse-reward navigation, how this approach can be used to facilitate better generalization, and at the same time, try to find more powerful theories that guide us better.*

Reviewer comment (Reproducibility):
There is very little information about the way experiments were carried out, particularly with regards to hyperparameter tuning for the proposed algorithm and for the baselines on the various environments. These details are required to assess the validity of the claims and to assure reproducibility.

Author response:
* "How were hyperparameters tuned for the algorithm (the algorithmic specific ones, such as timeout, pruning threshold, k for k-medoid, etc. as well as the ones basis distributed DDQN) and the various baselines for this task?"

***The following blocks are condensed and added to the revised manuscript:***

***Timeout and Pruning Threshold***

> Intuitively, we set the timeout to be equal to the distance pruning threshold. The timeout kicks in when the agent thinks a checkpoint can be achieved within e.g. 8 steps, but already spent 8 steps yet still cannot achieve it.

> This leads to how we tuned the pruning (distance) threshold: we fully used the advantage of our experiments on a fully DP-solvable environment: with a snapshot of the agent during its training, we can sample many <starting state, target state> pairs and calculate the ground truth distance between the pair, as well as the failure rate of reaching from the starting state to the target state given the current policy \pi, which is again enabled by DP. Then, we plot them as the x and y values respectively for visualization. We found such curves to evolve from flat high failure rate with all distances at the beginning, to a monotonically increasing curve (w.r.t. the true distances), where at small true distances, the failure rates are near zero. We picked 8 because the curve starts to grow explosively when the true distances are more than 9.

*These visualizations were key validations for the effectiveness of our proposed estimators (we do the same validations for other estimators as well). Our architectures (depths, widths of MLPs, size of distributional outputs) on top of the baseline distributional DDQN were iterated (minimally) to achieve desired results on these visualizations while minimizing the computational demand.*

*We compiled a gif for you to visualize how the error in the estimation of the reward estimator evolve during training, which can be accessed here:*

*https://github.com/AnonymousAuthors21/ICLR2024-Rebuttal/blob/main/evolution_reward_error.gif*

***K for k medoids***

*We tuned this by running a sensitivity analysis on skipper agents with different k’s. The results are in the Appendix B.6.*

*Additionally, we prune from 32 checkpoints because 32 checkpoints could achieve an ok coverage of the state space (maximally 12^2=144 states) as well as its friendliness to NVIDIA accelerators (batch size 32 is fast).*

***Size of local Perception Field*** 

*We used a local perception field of size 8 because our baseline model-free agent would be able to solve and generalize well within an 8x8 task but not larger. Roughly speaking, our spatial abstraction breaks down the overall tasks into 8x8 sub-tasks, which the policy could comfortably solve.*

*The local perception field is implemented in its simplest form for vision-based tasks: an 8x8 convolutional layer (no padding) on top of the embedded input.*

***Model-free Baseline Architecture***

*The baseline architecture (distributional, Double DQN) was heavily influenced by the architecture used in (Zhao et al., 2021), which demonstrated success on similar but smaller-scale experiments (8x8). The difference is that while they used computationally heavy components such as transformer layers on a set-based representation, we replaced them with a simpler and effective local perception component, as discussed just now. We validated successfully our model-free baseline performance on the tasks proposed in (Zhao et al., 2021).*

*No training-related hyperparameters were changed in this work, which include the intervals of training, the learning rate, the optimizer, etc. These are in fact identical to those in Rainbow (Hessel et al., 2017).*

===

***Finally, we hope that our answers addressed your concerns well and have improved the manuscript considerably. We would greatly appreciate it if you could consider increasing your rating of this paper, as it matters a lot to us! Thank you very much for your time!***

Reviewer comment (Evaluation):
While the experiments run with 20 seeds are a strength, in order to claim statistical significance it is suggested that after this pilot study, statistical power analysis should be performed to determine the necessary sample size. It is recommended to either add such a statistical power analysis to the paper or soften the claims accordingly.

Author response:
* "I listed the experiments ran with 20 seeds as strength and it is. However, to claim significance, it is suggested that after the pilot study with 20 seeds, statistical power analysis should be performed to determine the necessary sample size [1][2].
[1] How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments, Cedric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer, 2018
[2] Empirical Design in Reinforcement Learning, Andrew Pattersion, Samuel Neumann, Martha White, Adam White, 2023 
I recommend either adding a statistical power analysis to the paper or softening the claims a little bit."

*Thank you very much for your constructive suggestions. We have invested effort to conduct pairwise Welch’s t-tests (for better claims of statistical significance with 20 seed runs) as well as statistical power analyses for all training configurations. The tables below (via urls) show the results (established over the final performance of each run):*

*In the first table, we are showing the pairwise comparison between Skipper-once and the other methods. For each cell of the table, NO is presented if the pairwise t-significance is lower than set threshold (alpha = 0.05) and otherwise, the minimum number of independent seed runs that achieves statistical power (beta < 0.2) is reported. The insignificant results are marked in bold.*

*https://github.com/AnonymousAuthors21/ICLR2024-Rebuttal/blob/main/power_once.png*

*Similarly, here is the table for Skipper-regen v.s. other methods:*

*https://github.com/AnonymousAuthors21/ICLR2024-Rebuttal/blob/main/power_regen.png*

*Based on this new evidence, we have softened our claims of significance accordingly in the revised manuscript (still, for 25, 50, 100 training environments, the results are significant and powerful).*

*Additionally, we have added acknowledging citations to the two suggested references and have added the tables and related discussions to the revised Appendix. Thank you again for the constructive improvement that your comments have brought to this revision.*

Reviewer comment (Writing):
The abbreviation RL should be defined as “reinforcement learning (RL)” the first time it is used. Even though it is clear for the audience, it is good practice.

Author response:
* "The abbreviation RL should be defined as reinforcement learning (RL) the first time it is used., even though it is clear for the audience what it stands for, it is good practice."

*Thank you very much for your suggestion. We have adjusted the revised manuscript accordingly.*

Reviewer comment (Writing):
In Section 2, 2nd paragraph, and Section 4, 1st paragraph, there is no need to capitalize the word following a semicolon (;). Alternatively, end the previous subsentence with a full stop and then capitalize the start of the sentence.

Author response:
* "Section 2, 2nd paragraph, also Section 4 1s paragraph: no need to capitalize the word following a semicolon (;) or end the previous sub-sentence with a full stop and the capitalize the start of the sentence." 

*Thank you very much for your suggestion. We have adjusted the revised manuscript accordingly.*

*We addressed the capitalization problem not only in the suggested paragraphs but also checked globally that such problems no longer exist.*

Reviewer comment (Writing):
In Section 2, 2nd paragraph: The sentence “evaluated _on_ unseen tasks, _in which_ there are environment variations, but the core strategies need to finish the task remain [...]” should be corrected to use "remain" → "remains".

Author response:
* "2nd paragraph: "evaluated on unseen tasks, in which there are environment variations, but the core strategies need to finish the task remain [...]" (not remains)"

*Thank you very much for your suggestion. We have adjusted the revised manuscript accordingly.*

Reviewer comment (Writing):
In Section 2, 3rd paragraph: The phrase “Recent work suggests that this approach” should be replaced with “the latter approach”.

Author response:
* "3rd paragraph: "Recent work suggests that this approach" -> the latter approach"

*We appreciate your suggestion. We have adjusted the revised manuscript accordingly.*

Reviewer comment (Presentation):
Figure 2 should be moved at least one page down, as it contains concepts that are explained much later in the paper.

Author response:
* "Figure 2 should be moved at least one page down, there are concepts in it that are explained much later."

*Addressed by moving the figure location according to the suggestion.*

Reviewer comment (Writing):
In the sentence “The following estimates are learned with using distributional RL” the wording should be corrected to either “with” or “using” but not both.

Author response:
* " 'The following estimates are learned with using distributional RL' - with or using"

*Thank you very much for your suggestion. We have adjusted the revised manuscript accordingly.*

Reviewer comment (Writing):
In Section 3.3, the expression “on top of” occurs twice; it is rather ambiguous. The reviewer suggests replacing it with something clearer, such as “with” or “after”, or another expression conveying the intended meaning.

Author response:
* "the expression "on top of" occurs twice and I find it rather ambiguous. Could you use some other expression, e.g. with or after, or anything else that conveys what you are trying to say."

*Thank you very much for your suggestion. We have adjusted the revised manuscript accordingly.* 

*Occurrence regarding the usage of the partial state extractor, changed to:* 

> Through $\sigma$, the auxiliary estimators, to be discussed soon, force the bottleneck mechanism to promote aspects relevant to the local estimation of connections between the checkpoints.

*Occurrence regarding the usage of partial state representation, changed to:*

> The rewards and discounts are then estimated from the partial state $\sigma(S)$, based on the agent's behavior.

*Occurrence regarding the addition of a loss signal, changed to:*

> This is implemented trivially by adding a loss to the original training loss for the distance estimator, which we give a $0.25$ scaling for stability.

*Also changed the occurrences in the Appendix.*

Reviewer comment (Writing):
In Section 3.3.2, 3rd paragraph: The sentence “Similarly to $V_{\pi}$, we would want to know the cumulative discount leading to the target s⊙ under $\pi$.” is unnecessary. The following sentence could simply start with “The cumulative discount leading to the target s⊙ under $\pi$ is difficult to learn…”.

Author response:
* "3.3.2 3rd paragraph: "Similarly to ��, we would want to know the cumulative discount leading to the target s⊙ under �." - There is no need for this sentence. Just start the next sentence as 'The cumulative discount leading to the target s⊙ under � is difficult to learn...'"

*Thank you very much for your suggestion. We have adjusted the revised manuscript accordingly.* 

*Changed to:* 

> The cumulative discount leading to the target $s_\odot$ under $\pi$ is unfortunately more difficult to learn than $V_\pi$, since the prediction would be heavily skewed towards $1$ if $\gamma \approx 1$.

Reviewer comment (Writing):
In Section 3.4, 2nd paragraph: The phrase “tall order” is slang; it is suggested to replace it with a clearer term, such as “difficult task”.

Author response:
* "3.4 second paragraph: "tall order" - this is slang, consider changing it to something like 'difficult task' "

*Thank you very much for your suggestion. We have adjusted the revised manuscript accordingly.* 

*Changed to “challenge”.*