Paper: Bi-Directional Goal-Conditioning on Single Policy Function for State Space Search (ICLR-2024)

Reviewer comment (Experiments):
The experimental results presented in Figure 3 and 4 are not convincing at all. The tasks `NChain` and `Simple GridWorld` are too simple and despite that the results seem very unstable.

Author response:
**1. Experimental Results' Robustness (Figures 3 and 4):**
We acknowledge your concern regarding the robustness of our experimental results. HER and other methods used here are well-established algorithms that can solve this state space search problem (which is a simple MDP) very well. Due to this reason, we’ve not added a best effort implementation. When all variants perform well, it might become hard to explicitly understand the performance of agents when using samples from a bigger task space. We observe this in Fig 3. NChain experiments. To differentiate the performances better, we use these experiment parameters (function approximator size, number of updates to the UVFA, exploration parameters, number of episodes and so on..) borrowed from NChain experiments to varying complexities of GridWorld. Fortunately, we were able to see clear differences in performance of the models and have reported the results. The code will also be made available under anonymity and a link to the same will be added to the abstract of the updated paper. Thank you for your suggestion on changing the experiment from single-seeded to multi-seeded results, we will modify our results based on this and update our results.

Reviewer comment (Reproducibility):
All the results are single-seeded and there is no measure of variance among multiple runs. I recommend the authors to present multi-seeded results over at least `10` random seeds per run to ensure reproducibility.

Author response:
**1. Experimental Results' Robustness (Figures 3 and 4):**
We acknowledge your concern regarding the robustness of our experimental results. HER and other methods used here are well-established algorithms that can solve this state space search problem (which is a simple MDP) very well. Due to this reason, we’ve not added a best effort implementation. When all variants perform well, it might become hard to explicitly understand the performance of agents when using samples from a bigger task space. We observe this in Fig 3. NChain experiments. To differentiate the performances better, we use these experiment parameters (function approximator size, number of updates to the UVFA, exploration parameters, number of episodes and so on..) borrowed from NChain experiments to varying complexities of GridWorld. Fortunately, we were able to see clear differences in performance of the models and have reported the results. The code will also be made available under anonymity and a link to the same will be added to the abstract of the updated paper. Thank you for your suggestion on changing the experiment from single-seeded to multi-seeded results, we will modify our results based on this and update our results.

Reviewer comment (Evaluation):
`Figure 3`: All methods including the `HER` and `RE` baselines seem to work on-par and `SRE_NITR` diverges on the 10Chain. And on 15Chain all methods perform the same.

Author response:
**1. Experimental Results' Robustness (Figures 3 and 4):**
We acknowledge your concern regarding the robustness of our experimental results. HER and other methods used here are well-established algorithms that can solve this state space search problem (which is a simple MDP) very well. Due to this reason, we’ve not added a best effort implementation. When all variants perform well, it might become hard to explicitly understand the performance of agents when using samples from a bigger task space. We observe this in Fig 3. NChain experiments. To differentiate the performances better, we use these experiment parameters (function approximator size, number of updates to the UVFA, exploration parameters, number of episodes and so on..) borrowed from NChain experiments to varying complexities of GridWorld. Fortunately, we were able to see clear differences in performance of the models and have reported the results. The code will also be made available under anonymity and a link to the same will be added to the abstract of the updated paper. Thank you for your suggestion on changing the experiment from single-seeded to multi-seeded results, we will modify our results based on this and update our results.

Reviewer comment (Evaluation):
`Figure 4`: Mixed and unstable results. The proposed method `SRE_NC` seems to have divergence issues.

Author response:
**1. Experimental Results' Robustness (Figures 3 and 4):**
We acknowledge your concern regarding the robustness of our experimental results. HER and other methods used here are well-established algorithms that can solve this state space search problem (which is a simple MDP) very well. Due to this reason, we’ve not added a best effort implementation. When all variants perform well, it might become hard to explicitly understand the performance of agents when using samples from a bigger task space. We observe this in Fig 3. NChain experiments. To differentiate the performances better, we use these experiment parameters (function approximator size, number of updates to the UVFA, exploration parameters, number of episodes and so on..) borrowed from NChain experiments to varying complexities of GridWorld. Fortunately, we were able to see clear differences in performance of the models and have reported the results. The code will also be made available under anonymity and a link to the same will be added to the abstract of the updated paper. Thank you for your suggestion on changing the experiment from single-seeded to multi-seeded results, we will modify our results based on this and update our results.

Reviewer comment (Experiments):
There has been more work for exploration in RL, looking at expanding trees and search - for example [1] which can potentially be a baseline.

Author response:
**2. More Exploration Settings & Planning Literature**
Thank you for pointing out some interesting literature. The primary research focus of our work hasn’t been to come up with the most optimal exploration/planning setting for state space search problems, but instead, just to check if defining new tasks from an existing task (if the original/forward task is to go from a start state to a goal state) like backward or intermediate tasks (start from goal state and go to the original start state/ or an intermediate state to another intermediate state) and overload all these tasks on a single policy function (by extending GCRL) helps the agent perform better in the original task. We’ve designed our baselines in a manner to exactly understand how different task definitions & formulations and training affects the performance in the main task. 

Thank you for pointing us to multiple exploration ideas. As part of our future work, we plan on extending this way of generating new tasks and training the agent, by combining it with various existing exploration and planning ideas.

Reviewer comment (Writing):
The writing and flow of the related work can be improved significantly. For instance, there is a related work section on `Surrogate Objective for Goal-conditioned RL` going through equations for the Lipschitz continuity assumption and Wassersetein distance and it’s not even relevant to their framework. Overall, this makes reading the paper more difficult because I don't need to see the equations for related work.

Author response:
**3. Writing and Structure of Related Work:**
We agree that the clarity of the related work section can be improved. We have revised this section, focusing on directly relevant literature and removing unnecessary technical details, such as the equations for the Lipschitz continuity assumption and Wasserstein distance. This revision would definitely make the paper more accessible and focused on our framework.

Reviewer comment (Experiments):
Did the authors try other exploration mechanisms apart from `ϵ-greedy`? For instance `Curiosity-driven Exploration`, or `Max-Entropy-RL` methods?

Author response:
**2. More Exploration Settings & Planning Literature**
Thank you for pointing out some interesting literature. The primary research focus of our work hasn’t been to come up with the most optimal exploration/planning setting for state space search problems, but instead, just to check if defining new tasks from an existing task (if the original/forward task is to go from a start state to a goal state) like backward or intermediate tasks (start from goal state and go to the original start state/ or an intermediate state to another intermediate state) and overload all these tasks on a single policy function (by extending GCRL) helps the agent perform better in the original task. We’ve designed our baselines in a manner to exactly understand how different task definitions & formulations and training affects the performance in the main task. 

Thank you for pointing us to multiple exploration ideas. As part of our future work, we plan on extending this way of generating new tasks and training the agent, by combining it with various existing exploration and planning ideas.

Reviewer comment (Experiments):
I believe the biggest weakness is that the empirical evaluation is done in simple environments, NChain and GridWorld. I believe that GCRL, which is about overcoming difficulties in reaching different goal states, needs environments with complexities in their dynamics to some degree (e.g., locomotion environments from MuJoCo) as testbeds.

Author response:
We build our idea on a fundamental assumption that if there is a path existing from start state to goal state, there is also a path back from the goal state to the start state. We tried to exploit the strength of GCRL, to train an agent to go from any state to any state, which implicitly leads to an assumption of action reversibility for each state-action pair. Unfortunately, this doesn’t hold true for typical RL benchmark environments. Due to this reason, we stick to typical state space search problems (problems like NChain and GridWorld)

We agree that these environments (problems like NChain and GridWorld) used in our empirical evaluation are relatively simple. However, these simpler environments satisfy our reversibility assumption and also help us demonstrate our hypothesis of multi-task (start state -> goal state, goal state -> start state, intermediate states -> goal state and so on) definition learning and compare it with linearly increasing problem complexity. We acknowledge the need to test our approach in more complex environments, such as MuJoCo's locomotion tasks, which could better highlight the challenges and advantages in combining multi-task reformulations with GCRL. In future work, we plan to extend our evaluation to these more complex scenarios (that still remain reversible) to further validate the effectiveness of our approach.

Reviewer comment (Experiments):
I have some concerns about the main assumption of the ability to spawn agents at arbitrary states, especially in the GCRL setting, where *reaching* specific goal states is the objective. Taking advantage of the simulated environments, if spawning at arbitrary states can be done without any costs, some combination of local exploration and spawning might be effective for both exploration and gathering various samples for re-labeling and goal-conditioned training.

Author response:
It is true that the ability to spawning at arbitrary states when combined with the GCRL setting will definitely improve the quality of samples obtained (after relabelling) and this is what we wish to fundamentally explore. Although, to prevent any unplanned correlations between the task definitions, we don’t allow the agents to explicitly remember/control the spawning state sets. 

Our primary research objective in this work has been to check, if the multiple different tasks defined during the exploration phase (starting from any state s’ and reaching new state s’’) generating more samples with rewards, enables the agent to learn more about the whole environment better, and if this knowledge will help in solving the primary task (start state and goal state of the original state space search problem) and from our set of experiments, there seem to be some potential improvement which deserves further and in-depth study. We believe this ideation and formulation of the GCRL problem will be further broken-down and studied by the community.

Reviewer comment (Theory):
It seems as if there is an implicit assumption (which should be explicitly stated) that the environment is reversible. That is, the start states can always be reachable from the goals states. This is not always true, for example you can't "uncook" food.

Author response:
**Assumption of Environment Reversibility**

Our main assumption would be that there exists a path between goal state and start state (essentially, every new task we define, there exists a path from the start state and goal state of that particular task). The point you’ve mentioned is very true, since as we increase the number of tasks, our assumption does directly converge to reversibility of state-action pairs. We acknowledge that we did not explicitly state this assumption. Thank you very much for pointing that the point is missing and the necessity to make it clear and explicit. We will add a small paragraph to discuss this limitation & assumption and make it explicit. We will also add a small part explaining how, these specific MDPs where path from any state to another and the goal state set is a subset of the original state space, are what we call ‘state space search problems’ in our work.

Reviewer comment (Experiments):
All experiments are performed on extremely simple toy MDPs where judging the benefits of a complex goal sampling technique can be difficult. Moreover, the proposed method SRE-DQN doesn't perform the best in any of the tasks.

Author response:
**Environments, Performance of SRE-DQN not being the best & relative poor performance of other agents:**

We agree that a broader range of experiments is necessary. We agree that these environments (problems like NChain and GridWorld) used in our empirical evaluation are relatively simple. However, these simpler environments satisfy our reversibility assumption and also help us demonstrate our hypothesis of multi-task (start state -> goal state, goal state -> start state, intermediate states -> goal state and so on) definition learning and compare it with linearly increasing problem complexity. 

We acknowledge the need to test our approach in more complex environments, such as MuJoCo's locomotion tasks, which could better highlight the challenges and advantages in combining multi-task reformulations with GCRL. In future work, we plan to extend our evaluation to these more complex scenarios (that still remain reversible) to further validate the effectiveness of our approach.

**Regarding SRE’s Performance:**

Our primary research focus in this paper has been about generating new ‘task’ definitions within the same state space, training the model on these newly defined tasks, and seeing if that helps the agent in solving the primary state space better. The full SRE agent uses forward, backward and multiple intermediate task definitions (multi-directional). Following the main claim of the paper (bi-directionality/two-task definitions), we see that SRE_NC (variant that only uses bi-directionality and does not use intermediate candidates) seems to be working well for our experiment setting.

Reviewer comment (Reproducibility):
It is surprising that the success rate on such simple MDPs is lower than 20%. For example in a 3x3 MDP, both the SRE-DQN and HER get around 0 success rate. I suspect that an error in the implementation is causing this.

Author response:
**Regarding Low Success Rates:**

As you have mentioned, HER and other methods used here are well-established algorithms that can solve this state space search problem (which is a simple MDP) very well. Due to this reason, we’ve not added a best effort implementation. When all variants perform well, it might become hard to explicitly understand the performance of agents when using samples from a bigger task space. We observe this in Fig 3. NChain experiments. To differentiate the performances better, we use these experiment parameters (function approximator size, number of updates to the UVFA, exploration parameters, number of episodes and so on..) borrowed from NChain experiments to varying complexities of GridWorld. Fortunately, we were able to see clear differences in performance of the models and have reported the results. The code will also be made available under anonymity and a link to the same will be added to the abstract of the updated paper.

Reviewer comment (Reproducibility):
Implementation details as well as the code have not been provided.

Author response:
The code will also be made available under anonymity and a link to the same will be added to the abstract of the updated paper.