From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper presents an idea for using symbolic model learning to aid in exploration to find an optimal plan in goal-directed sequential decision making. The idea is to start with an overcomplete/optimistic model estimate, and then to refine/prune it as experience is gathered. Results are presented in several classical planning domains (e.g. blocksworld) as well as symbolic forms of RL-focused domains (e.g. minigrid, kitchen), showing that the proposed method outperforms various RL methods (Q-learning, HRL) and a symbolic baseline. While the central idea behind the paper is quite interesting and worthy of attention, the paper in its current form is not ready for publication at ICLR. While I and the reviewers appreciated the addition of the kitchen experiments and symbolic baseline, there was agreement that (1) the paper was difficult to understand, and would not be replicable as written; (2) the Q-learning baseline was inappropriate as it is not a method for exploration---something like Go-Explore (Ecoffet et al. 2021) would be more appropriate; and (3) a central claim of the paper is that the method is applicable to RL, but the paper does not actually demonstrate this empirically (e.g. by training an RL agent on the data generated by the symbolic exploration). As such, I recommend rejection.
justification_for_why_not_higher_score: While an interesting idea, the paper is not written clearly enough for the method to be useful to the community. It also has a number of other limitations, such as inappropriate baselines and unsupported claims.
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: I held an AC/reviewer meeting as the scores of the reviewers were split and the reviewers had not responded to the authors. However, in the meeting it became apparent that despite the author response, all reviewers (including those who had given a positive score) had reasonable concerns with the paper. The biggest concern was the clarity of the paper, as all of us had difficulty understanding exactly how the method worked; this concern, while raised by the reviewers in their reviews, was not adequately addressed by the authors in their response. Other concerns included the adequacy of the baselines (which was partially addressed by the authors by including some additional baselines, but not fully as some of the baselines included were still felt to be inappropriate) and the applicability to RL (which was  clarified by the authors in their response, but also made clear that the method is only in principle applicable to RL and has not been demonstrated empirically in the paper). My biggest concern is the clarity of the paper; if it is not clear what was actually done, then it is of limited value to the scientific community, and this (along with the other concerns) leads me to my decision to reject.

From: ICLR.cc/2023/Conference/Paper6083/Authors
title: Follow up
comment: Dear Reviewer m6BR,
Thank you again for your thoughtful reviews. As you may have seen, we have provided an updated version of the paper that now includes a more expanded evaluation section. We have now included the sample count information and the information on the number of samples used by the HRL baseline, among other changes. Please let us know if our reply and modifications to the paper did not alleviate your concerns. Feel free to reach out if you have any further questions.

Authors

From: ICLR.cc/2023/Conference/Paper6083/Authors
title: Follow up
comment: Dear Reviewer dzDP,

We would again like to thank the reviewers for their constructive comments. We have followed your comments and updated the paper, including adding a new baseline and providing extended discussion on online vs offline model learning. Please let us know if our reply and modifications to the paper did not alleviate your concerns. Feel free to reach out if you have any further questions.

Authors

From: ICLR.cc/2023/Conference/Paper6083/Authors
title: Follow up
comment: Dear Reviewer mjER,

Thank you for taking the time to providing the helpful comments. As you might have seen, we have updated the paper and provided a few clarifications for some of your questions. We were wondering if you had a chance to look at them. Please let us know if our reply and modifications to the paper did not alleviate your concerns. Feel free to reach out if you have any further questions.

Authors

From: ICLR.cc/2023/Conference/Paper6083/Authors
title: Follow up
comment: Dear Reviewer DcwG,

Thank you again for your thoughtful review. We had updated the paper in accordance with your recommendation and we even included the kitchen domain in the new set of results. We were wondering if you had a chance to look at the updated version and our response provided earlier. Please let us know if our reply and modifications to the paper did not alleviate your concerns. Feel free to reach out if you have any further questions.

Authors

From: ICLR.cc/2023/Conference/Paper6083/Authors
title: Response to Reviewer m6BR
comment: We thank the reviewer for their comments. Below are our responses to some specific questions the reviewer had raised

HRL Baseline: We do include an HRL baseline, the work [1] that we used as a point of comparison for Minigrid was an HRL baseline. We did not include it originally in the table because it had timed out. However, in the updated version you can see the number of samples used by that method. In particular, it used on average 188416.8 and 106821.4 samples for each of the two problems.

Sample Count: While we included the number of samples collected for each of the methods that succeeded, we agree that it would have helped to include the number of samples collected on all instances that didn’t complete. We have now included that information in the newest revision. Especially for the Q learning baseline, we can see that in general it took more samples than our method.


[1] Lee, Junkyu, et al. "AI Planning Annotation for Sample Efficient Reinforcement Learning." arXiv preprint arXiv:2203.00669 (2022).

From: ICLR.cc/2023/Conference/Paper6083/Authors
title: Response to Reviewer dzDP
comment: We thank the reviewer for their comments. We updated our draft to reflect those comments. In particular, the discussion related to specific functions discussed in the pseudocode are included in Section 4.2 and the potential interaction with the Q learning algorithm is provided at the end of the same section. Below we have provided some responses to specific questions raised.

Interaction between these Models and RL Algorithm: We see the primary use of these methods to be as a way to initialize the Qvalue function estimates, which can then be improved by the RL algorithm over time. Additionally, after the initialization, we can also employ it during the learning process like a traditional exploration strategy (note that in these later deployments the model will start from a more informative estimate of the model). 

Other Online Model Learning Work: We only found two other online model learning works, namely [1] and [2] that tries to learn a complete planning model (we have included a discussion about this work in the new draft). Works like [3] only learn some components of the overall model and thus are not suitable for our purposes. The main difference between [1],[2] and our work is the underlying objective of the learning process. While the objective of that work (and in fact most of the other model learning work) is to accurately reconstruct the underlying model, ours is to merely retrieve a path to the goal. This is because we are employing it purely as a means of driving the exploration. This brings up new challenges and opportunities that are unique to our setting.  For example, we are unaware of any other methods that try to maintain an optimistic approximation of the underlying planning model. Additionally, it is worth noting that the problem of building a planning model to drive exploration, in itself involves balancing exploration and exploitation. After all, we would want to be cognizant of what actions we test as part of a potential plan to the goal (and how often we test them).

Symbolic Baseline: We have now included a symbolic baseline to the current set of experiments. As discussed in our response to reviewer DcWG, in the one problem the baseline method was able to handle, the baseline required almost 900 times more samples than out method.

Hierarchical RL: One of the specific advantages our method provides over traditional hierarchical RL methods is that our method allows us to perform goal directed exploration. While the abstractions provide some structure to the underlying problem, the meta-controller is still limited by the need to perform blind exploration using the temporal extended actions.

[1] Lamanna, Leonardo, et al. "Online Learning of Action Models for PDDL Planning." IJCAI. 2021.

[2] Jaime G Carbonell and Yolanda Gil. Learning by experimentation: The operator refinement method. In Machine learning, pp. 191–213. Elsevier, 1990.

[3] Sreedharan, Sarath, et al. "Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representations." arXiv preprint arXiv:2002.01080 (2020).

From: ICLR.cc/2023/Conference/Paper6083/Authors
title: Response to mjER
comment: We thank the reviewer for their constructive comments. We have tried to address the comments that the reviewer had raised about the clarity of the paper. 
One of the reviewer’s comments was not clear to us: “This paper also relies on human input and a goal directed planner, those two assumptions do not seem realistic to solve more general problems.”
We would appreciate it if the reviewer could clarify what assumptions about a goal-directed planner would be unrealistic. 
Below, we have tried to respond to some specific comments the reviewer had

Deterministic Domains: As pointed out in our response to reviewer DcWG, our method remains a viable technique in stochastic settings. Please see the answer to reviewer DcWG for additional details.
It is not clear that stochasticity is one of the main causes of difficulty of exploration in reinforcement learning in sparse reward domains. Evidently, the exploration remains hard even in the absence of stochasticity. One foundational problem here is large state spaces and lack of any guidance to drive the agent towards high reward states. In order to make the experimental setting clean, we have decided to focus on the deterministic environments.

Blocksworld: Apart from the historical importance of the problem, blockworld is one of the simplest problems that really showcase the combinatorial explosion that occurs when considering a relational or object oriented representation of the problem. In fact, for a relatively small number of blocks (just above 100), the state space is larger than chess or Go (3.86E+172 states for a problem with 104 blocks). It is also worth noting that optimal planning for blocksworld is an NP-Hard problem. The reviewer can refer to [2] for a more detailed discussion on the importance of the blocksworld as a useful benchmark for sequential decision-making problems. 

Assumption made about Human Input: Please note that use of human feedback is a common assumption made within many reinforcement learning methods that leverage human-feedback. However, we would argue that the assumptions made in this work are much more reasonable than the assumptions made by existing works in this direction. Most of the prominent works in this direction expects a human observer to watch the RL agent execution and then provide feedback with respect to the agent behavior (cf. [3] and [4]). We, on the other hand, are only asking the human to provide a list of objects that may be relevant to a given action (please see an example in the response to reviewer DcWG). 
Even in that case, our method allows the human to provide a superset of objects. In fact, for domains with a bounded number of objects we can in theory start with the set of all objects, not requiring explicit input from the human.

Non-Tabular Q learning: Given the sparse reward settings and the fact that the actor is restricted to starting episodes from a specific initial state, Q learning based methods, regardless of whether it is tabular or using some form of function approximation, will be limited by its inability to generate useful Q value estimates. Since one can’t build any useful Q value approximations until a goal state is reached, which constitutes the entirety of our exploration problem.


[1] Camacho, Eduardo F., and Carlos Bordons Alba. Model predictive control. Springer science & business media, 2013.

[2] Slaney, John, and Sylvie Thiébaux. "Blocks world revisited." Artificial Intelligence 125.1-2 (2001): 119-153.

[3] Christiano, Paul F., et al. "Deep reinforcement learning from human preferences." Advances in neural information processing systems 30 (2017).

[4] Warnell, Garrett, et al. "Deep tamer: Interactive agent shaping in high-dimensional state spaces." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

From: ICLR.cc/2023/Conference/Paper6083/Authors
title: Response to Reviewer DcwG
comment: We thank the reviewer for their comments. Below we have provided responses to some specific questions the reviewer had raised

Deterministic Settings: As discussed in the paper, our basic method is directly applicable in stochastic settings, by assuming that the learned model is a determinization of the underlying true stochastic problem. Thus we can still maintain a deterministic version of the task, which is easier to plan with and the action sequences generated that way can then be used to drive the exploration in the true model. Such a determinization of models is a common technique used to simplify planning in works from both the symbolic planning community (eg: [1]) and control-theoretic planning literature (certainty equivalence is a core concept within model-predictive controllers [2]). In fact, such model determinization can be thought of as another way we are creating optimistic estimates of the underlying model.

Online vs Offline Model Learning: One of the big reasons we chose to focus on the online settings, is because it allows the planning techniques to leverage an important factor that has fueled recent successes in RL, namely the availability of powerful simulators. The availability of such simulators combined with access to efficient diverse planners allows us to generate goal reaching traces with high confidence, even when our model may be a loose approximation of the true underlying model. 

Symbolic Baseline: We have now included a baseline (one presented in [3]) that uses traces generated through random walks from the model to learn the symbolic model (provided in A.4.1). We specifically tested on the blocksworld domain with a budget of 600000 samples, where we found that the method was only able to learn a model capable of generating a valid plan for the first problem instance and required on average 548287.8 (averaged across five runs). This is nearly 900 times larger than the sample count required by our method on the same problem.

Kitchen Domain: We have now included an experiment on the symbolic part of the kitchen domain (provided in A.4.2). Under this domain, each action corresponds to the achievement of one of the highlevel subgoals. The relation between the various subgoals were encoded as per the description provided by Xu et al. 2019. For the experiments, we used a symbolic simulator, but as discussed by Xu et al. 2019, we can use a motion level planner to test the executability of each of these actions. The lifted version of our method was able to identify a valid plan in 61.46 sec using 24727.2 time steps (averaged across five runs) and the non lifted version took 393.57 secs and used 140681 samples (again averaged across five runs).

Human Feedback: The input we request from the human is very limited: a list of possible objects that might be related to a given action. For example, in the case of the action for unstacking block1 from block2, all the human has to identify is that action will only involve the blocks block1 and block2. It is also worth mentioning that access to this information is a common assumption made across existing works for learning symbolic models. We are unaware of any current RL methods that learn from such sparse feedback, all of the ones we are aware of usually require more involved feedback (for example [4],[5],and [6]).

[1] Yoon, Sung Wook, Alan Fern, and Robert Givan. "FF-Replan: A Baseline for Probabilistic Planning." ICAPS. Vol. 7. 2007.

[2] Camacho, Eduardo F., and Carlos Bordons Alba. Model predictive control. Springer science & business media, 2013.

[3] Brendan Juba and Roni Stern. Learning probably approximately complete and safe action models for stochastic worlds. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI, pp. 9795– 9804. AAAI Press, 2022.

[4] Christiano, Paul F., et al. "Deep reinforcement learning from human preferences." Advances in neural information processing systems 30 (2017).

[5] Warnell, Garrett, et al. "Deep tamer: Interactive agent shaping in high-dimensional state spaces." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

[6] Guan, Lin, Sarath Sreedharan, and Subbarao Kambhampati. "Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversity." arXiv preprint arXiv:2202.02886 (2022).

From: ICLR.cc/2023/Conference/Paper6083/Reviewer_DcwG
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper presents a new algorithm for integrating off-the-shelf automatic symbolic planners with online RL for efficient exploration in the sparse reward setting. The authors propose to itteratively refine a symbolic representation that a planner uses to predict paths towards a goal, which gets rolled out in the environment. It is theoretically proven that the proposed algorithm terminates, provided some conditions on the planner are satisfied. The authors show results comparing their method to Q-learning and show an improvement in the outlined domains. 
strength_and_weaknesses: Strengths:

The paper presents the problem setting and background in a very rigorous manner, and the theoretical guarantees are relevant to the proposed task. The paper includes code for reproduction.

Weaknesses:

The paper is lacking in thorough experimentation and results discussion. Additionally, a major limitation comes from the constraint to deterministic MDPs (which the authors acknowledge). Additionally, the paper assumes human input for annotation of the actions, which begs the question why the baseline is a Q-learning model, as opposed to something that similarly uses human guidance, i.e. imitation learning. Additionally, the use of online RL, as opposed to the more prevalent offline RL setting within the symbolic representations, is not well justified. It would be good to compare against other existing symbolic planning methods as well. 
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow and clearly written. The presented approach is novel although perhaps of limited practicality. The appendix contains sufficient implementation details for reproducibility.
summary_of_the_review: The paper is clear and well written with good theoretical justification, however it is questionable if the proposed approach could scale to more complex and realistic domains (such as the kitchen domain from Danfei Xu, Roberto Martín-Martín, De-An Huang, Yuke Zhu, Silvio Savarese, Li Fei-Fei, Regression Planning Networks). I would be willing to increase my score if more detailed experiments are provided.  
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6083/Reviewer_mjER
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes to learn a new exploration technique to learn a symbolic model of an environment. This model is then used to speed up goal directed exploration. The proposed technique is then evaluated against Q-learning with epsilon-greedy and R-max as well as a hierarchical RL method. Experiments are done on blockworld and minigrid and show that using the optimistic symbolic model improves results over the baselines
strength_and_weaknesses: While the paper proposes an interesting technique to solve the specifics problems presented in the paper it is unclear how applicable the proposed technique are to other problems.
For example the paper chooses to ignore the challenge of stochasticity in exploration in reinforcement which is one of the main cause of the difficulty of exploration in reinforcement learning
"We specifically chose a setting, which forefronts
the problems related to exploration, while placing less emphasis on other aspects of an RL problem".

This paper also relies on human input and a goal directed planner, those two assumptions do not seem realistic to solve more general problems.

I also do not understand why the paper decides to introduce and focus on blockworld. The paper does not properly justify why this is an interesting problem and we should care about solving as opposed to the larger number of existing reinforcement learning environments.

It appears that the paper is using a tabular version of Q-learning ? In that case can the proposed method scale to problems that require function approximation? In that setting it would be interesting to use more recent algorithms as baselines such as RND and/or NoveID.


clarity,_quality,_novelty_and_reproducibility: Overall I found that the paper was really difficult to follow.
Section 3.1 spends a lot of time to introduce the blacksworld environment even though the information presented is not particularly interesting and could be pushed to the appendix.

"In fact, possibly the only effective methods in the mainstream RL repertoire we can use are curiosity driven or intrinsic reward based methods" 
Posterior sampling methods such as Thompson sampling could be used.

"For the symbolic model, we will be using an untyped normalized PDDL task model Helmert (2009)."
Normalized PDDL were not introduced.
I found section 3.3 really hard to understand, a plot / figure of the symbolic model would have been helpful to help the reader.

"The basis of our approach is an observation that every deterministic MDP has a precise symbolic
representation." what does precise means?
summary_of_the_review: Overall while the proposed idea is interesting I do not think the paper present a compelling evidence that their method can be useful to improve exploration in most environments.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper6083/Reviewer_dzDP
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper focuses on the issue of requiring to access a complete and correct symbolic model when combining symbolic methods with reinforcement learning (RL). To address the issue, this work proposes a new method for learning optimistic symbolic approximations of the underlying model in an online fashion. Besides, theoretical results are provided to further analyze the proposed algorithm. Finally, experiments were conducted on three traditional planning domains (Blocks, Elevator, Gripper) and one traditional RL benchmark (Minigrid), with results demonstrating its effectiveness in terms of solvability, the average time, and the number of samples.
strength_and_weaknesses: Strengths:
- Symbolic representation has been shown promising in guiding RL for the efficient exploration, explainability, and human-machine interaction. This is an interesting work targets to learning a symbolic representation for effective exploration within RL, particularly under the sparse reward setting. It would be intriguing to both the communities of symbolic planning and RL.


Weaknesses:
- The presentation of the paper could be improved. For example, how the symbolic model in this work is learned in an online fashion? As mentioned in the related work, I notice some of them allowing to learn the symbolic representation in an online setting, what's the difference between the proposed method and the existing work? How human guidance/ advice interact with this symbolic representation in practice? There lacks more technical details. For the pseudocode of algorithm 1, it would be more clear if the procedures, such as PruneModel, UpdateModel, can be referred to a more specific steps/ equations/ definitions. Besides, I am wondering how does this learned representation interact with RL since I didn't find too many details about this in the paper. Also, how does this learned representation help RL w.r.t. the effective exploration? Does it aim to reduce the exploration space by using prior knowledge or guide RL to explore more meaningful states?
- Current empirical results could not well support the claims on addressing the problem of effective exploration and speeding up the learning process with human minimal input. Comparing with Q-learning with the proposed method (w/wo lifting) in Table 1 looks unfair to me. For the planning domains, there should be included some planning baseline in addition to the proposed method. Particularly, I am quite curious why hierarchical RL fails in Minigrid domain? Even if hierarchical RL doesn't work, it at least has a better performance than flat Q-learning. Can you elaborate more on this?


Minors:
- ’goal‘ on pp.3 should be 'goal'.

clarity,_quality,_novelty_and_reproducibility: - This paper could be improved by re-organizing the papers and showing more technical details as I mentioned in the main Weaknesses. Besides, there are some questions could be clarified.
- Novelty: this work provides some insights in the symbolic method side. However, it's unclear how it will help the RL side.

summary_of_the_review: According to my comments in both the main Weaknesses and the section of Clarity, Quality, Novelty, I feel this is a paper where reasons to reject outweigh reasons to accept.

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper6083/Reviewer_m6BR
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes to combine human input in the form of a set of predicates and objects that they associate with a given problem, and realising the fact that for every deterministic MDPs there is a symbolic representation that they dub optimistic representation that reaches the goal state. They provide  the theoretical underpinnings necessary to refine and update a model that will solve the task when combined with a complete planner.


strength_and_weaknesses: Strengths: This paper is clearly written and provides an interesting approach to Exploration by lifting the action representation to a symbolic level and combining exploration with planning. 
The approach seems very novel to the best of my knowledge.
Weaknesses: the experimental section is lacking in comparison with other HRL approaches and how is Q learning performing in the tasks, how far does it go or where does it fail? It would also help understand the significance of the contribution if the results were given in terms of samples as well.

clarity,_quality,_novelty_and_reproducibility: The paper introduces notation on an inverse mapping function that did not seem to be used later on, where was this lifting of the symbolic model being used? And how exactly is it being learned?
Suggestions:
The mapping between states and labels C(s) is applied to actions as well although it is defined over states can the authors clarify? What does it mean C(a) is a predicate true or false in the MDP ? when performing an action? But this is also dependent on the state where it is being executed /equivalent initial and seq of actions in a deterministic env.
The delete_{i+1} =del^a_{i} U (C(s’)\C(s)) reads equals to the del set union with the set of true predicates in (s’) set difference set of true predicates in s? It does not seem to match the text description that the set of predicates that are false in s’. It would help if the authors could clarify on the meaning of the operator \.
How is the model being pruned exactly? And how sensitive is the approach to the planner?
How sensitive is the optimistic representation to the size of the sets? Is the min_add max_del faster?
Typos:
-bee -> been pg 9

summary_of_the_review: This paper proposes a novel way of doing exploration using symbolic reasoning and combines it with planning. The main contributions seem to be in the model update by leveraging human input to formulate preconditions, deletion and addition of predicates for every state, and actions and using update rules to guarantee the existence of a path from start to goal conditions as long as a complete planner is in place. The experimental section only compares the proposed model with and without symbolic model lifting with Q learning in a time budget constraints. The paper would benefit if results would also be presented in terms of samples. Also no HRL baseline is provided, which would seem a more fair comparison.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold