Paper: Pylic: Leveraging Source Code for Planning in Structured Environments (ICLR-2024)

Reviewer comment (Novelty):
In my opinion, the paper seriously lacks novelty. It invents (or introduces) new terms for concepts that have long existed. E.g., the main contribution claimed by the paper is using **code of the simulator** for solving the problem faster. However, this is nothing but having access to an analytical model of the system. Why to complicate the paper? The second term would be the **meaningful events**. The events that are required or necessary to be achieved in order to reach the goal state. These is analogous to landmarks [1] or critical regions [2]. Landmarks and critical regions have been extensively used in planning and robotics literature.

Author response:
### On simulator code

We agree that source code is ultimately an analytical model. However, we argue that source code has key features that make it unlike other forms of analytical models. Specifically, our focus on leveraging control flow makes it possible to easily reuse parts of the model when communicating domain knowledge. Our claim is that this allows complex search spaces to be described efficiently by an expert user, allowing the system to find solutions to a problem domain, as shown in the experiments.

This is why the focus on source code is not inconsequential and is not equivalent to a general analytical model with no assumptions about its internal structure.

### On meaningful events

We appreciate the connection to landmarks and critical regions, and we agree that the concept of meaningful events is related to concepts used in other planning frameworks. Note that our "meaningful events", however, have additional assumptions specific to our framework, which would make the use of, e.g., the terms "landmarks" or "critical regions" incorrect (even if, broadly speaking, they are related). This is why we decided to use a different term. We have revised our manuscript to explicitly reference the related concepts.

## On "meaningful events", "landmarks," and "critical regions"

Our motivation for introducing the term "meaningful events" is two-fold:

- Central to our definition is the constraint that a "meaningful event" is something that a domain expert can describe as a "precise statement about the execution of the simulator[...]". With this definition, we intend to highlight two core assumptions of our framework: there is an expert user capable of relating domain knowledge to source code, and these statements relate to the execution of a structured model (simulator source code). Introducing the term "meaningful events" allows us to highlight these two non-trivial assumptions.

- Furthermore, we agree that "meaningful events" could be understood --e.g.,-- as a special case of "landmarks" with a "reasonable order". The key observation is that we are interested in describing properties about the execution of the model, not only about the simulated system.  We are then in (what would be) the special case where the set of atomic facts consists of statements about the execution of the structured model (further, note that this set is not finite) and the actions consist of a change over the control parameters. Thus, describing our approach from within the landmark framework would make the explanation unnecessarily verbose and non-intuitive, and introducing "meaningful events" as a self-contained concept leads to a more succinct description of our approach (note that we nonetheless updated the manuscript to acknowledge and cite these analogous concepts).

Because of these reasons, we argue, it is appropriate to introduce the term "meaningful events" for the purpose of describing the framework.

Reviewer comment (Novelty):
The approach requires the analytical of the model of the system as well as a set of landmarks or critical regions to be provided upfront. This does not only require a domain expert at the train time but at the test time as well. Which is infeasible to have. Especially, when a lot of research has been focused on learning these landmarks or critical regions automatically as well as approaches than learn policy without explicitly having access to an analytical model of the environment and treating the simulator as a blackbox.

Author response:
### On requiring a domain expert at test time
(Also included in response to Reviewer ppCk)

We want to clarify that the domain knowledge provided to the system is general to a problem domain and not bound to a particular problem instance. We believe this important point was not efficiently communicated in our manuscript. Specifically, the user provides general principles to search over sequences of meaningful events for an entire problem domain (section 3.2). We have revised our manuscript to clarify this.

Thus, we do not "require" the domain expert for any particular task; the expert is only required to describe the search over sequences of meaningful events for a problem domain.

## On expert at test time

You are correct in that, for different problems in the same domain, different sequences of meaningful events are needed to solve any particular task. However, note that the expert does not provide a particular sequence of meaningful events or even a fixed set of meaningful events. Rather, the expert describes how to construct sequences of meaningful events incrementally  ("GROW" routine in Algorithm 1). In this way, the system can search for sequences of meaningful events on its own.

In other words, the expert does not have to anticipate what a solution to any specific task would look like *exactly*; instead, it has to anticipate the *structure* of (possibly) useful sequences of meaningful events to incrementally search the solution space. Effectively, the expert is tasked with providing a single program ("GROW") for the entire problem domain that receives a candidate partial solution to any problem instance and answers the question, "Given *this* partial solution, what are some meaningful events the system could try solving for next?", which the system can then use to search over sequences by performing a tree search (a standard approach).

This avoids the problem of listing all possible meaningful events, which we agree is not feasible.

Reviewer comment (Writing):
It is not clear from the paper that how a sequence of low-level action is generated to reach each meaningful event. My educated guess is the problem is reduced to an optimization problem but it has to be clear from the paper.

Author response:
### On generating the sequence of low-level actions

Your assumption is correct; the problem is reduced to an optimization problem. This is described in Section 3.1 (now 3.2). We have revised the manuscript to make the description of the framework easier to follow.

Reviewer comment (Experiments):
Lastly, the empirical evaluation is extremely weak. Especially, the choice of the baselines. Given that this approach is a model-based optimization approach. This should be compared with a hierarchical planning approach [2,3,4] or a hierarchical optimization approach [5] or a model-based RL approach.

Author response:
### On the choice of baselines
(Also included in response to Reviewer wE34)

We agree that comparing our approach with pure RL and MPC approaches is not completely straightforward because we allow users to provide structured domain knowledge. We also agree that the techniques you list would be good candidates for further benchmarking.

Specifically, we are working on a comparison of our technique with planning techniques that leverage temporal specifications. This is more of an apples-to-apples comparison because some of these techniques also allow (and rely) on a human to provide the domain knowledge to make a domain of tasks feasible, and this knowledge is encoded at a comparable level of abstraction.

Reviewer comment (Theory):
The paper argues that other methods rely on specialized representations, which makes them hard or inconvenient to use. However, the proposed method requires the user to specify so-called "meaningful events". Judging from the examples and the description, this would appear to be an even more onerous requirement as they have to be defined by the end-user instead of the designer of a general method for particular problem scenarios. This aspect is insufficiently discussed in the paper, making it unclear that this approach is practical in contrast to other methods such as reinforcement learning, model-based control, or Monte Carlo tree search.

Author response:
As you correctly point out, our proposed approach currently relies on a human to provide the space of meaningful events for a given problem domain (and relate them to the source code). Thus, comparing our approach with pure RL and MPC approaches is not completely straightforward because we allow users to provide structured domain knowledge.

We are working on a comparison with planning techniques that leverage temporal specifications. This is more of an apples-to-apples comparison because some of these techniques also allow (and rely) on a human to provide the domain knowledge to make a domain of tasks feasible, and this knowledge is encoded at a comparable level of abstraction.

We argue that the amount of effort required by our approach is similar to, or smaller than, techniques that leverage domain knowledge during planning. This is because, in our framework, the user can encode complex conditions (e.g., a collision) simply by referencing the corresponding constructs in the meaningful event predicates without reimplementing them from scratch (provided that such constructs are present in the simulator, which is one of our assumptions).

We want to clarify that the domain knowledge provided to the system is general to a problem domain and not bound to a particular problem instance. We believe this important point was not efficiently communicated in our manuscript. Specifically, the user provides general principles to search over sequences of meaningful events for an entire problem domain (section 3.2). We have revised our manuscript to clarify this.

Reviewer comment (Experiments):
Tracing general program execution can still be interesting, as the experiments show. However, the description of the applications and properties described in the main body of the text is misleading. Without seeing the experiments, one would have expected the proposed method to be able to trace through complex physics engines as employed by Isaac Sim or pybullet.

Author response:
It is precisely the logic that reflects the structure of a problem domain that can be related to domain knowledge. As we show, this key logic can be low-level dynamics (e.g., collision handling), but more often it will be logic that leverages a low-level physics engine (e.g., the button dynamics in "Password locomotion"). This is why, in the experiments, we explain how we leverage code at the appropriate level of abstraction for a given problem domain.

Reviewer comment (Presentation):
The proposed method uses quite a few components and joins them together. While an overview is provided in Figure 1, this figure is never used in the text to help the reader understand how things connect. As such, it is hard to follow where the different pieces go and how they interact.

Author response:
The different components come together in Algorithm 2 (now Algorithm 1). We have revised the manuscript Methods section to make the description easier to follow and the connections between components more explicit.

Reviewer comment (Theory):
The paper states that the local search is sufficient to find parameters to reach the next meaningful event. However, it is not mentioned how this can be guaranteed or why this should hold in the first place.

Author response:
We assume that the user has provided events for which this assumption is true. We have revised our manuscript to further highlight this assumption.

In the special case where meaningful events induce convex problems, the connections between events can be guaranteed to be reachable with gradient descent. However, in general, the user is subject to the limitations of the optimization method used to define meaningful events.

Reviewer comment (Theory):
It is unclear whether there are theoretical guarantees that can alert the user when the local search approach is impossible, or if the user has to add "meaningful event" specifications until things can be solved.

Author response:
We assume that the user has provided events for which this assumption is true. We have revised our manuscript to further highlight this assumption.

In the special case where meaningful events induce convex problems, the connections between events can be guaranteed to be reachable with gradient descent. However, in general, the user is subject to the limitations of the optimization method used to define meaningful events.

Reviewer comment (Theory):
The reviewer questions whether the need to have traced control flow labels, predicates, and user-specified "meaningful events" is not more challenging and domain-specific than the representations required by other approaches.

Author response:
As you correctly point out, our proposed approach currently relies on a human to provide the space of meaningful events for a given problem domain (and relate them to the source code). Thus, comparing our approach with pure RL and MPC approaches is not completely straightforward because we allow users to provide structured domain knowledge.

We are working on a comparison with planning techniques that leverage temporal specifications. This is more of an apples-to-apples comparison because some of these techniques also allow (and rely) on a human to provide the domain knowledge to make a domain of tasks feasible, and this knowledge is encoded at a comparable level of abstraction.

We argue that the amount of effort required by our approach is similar to, or smaller than, techniques that leverage domain knowledge during planning. This is because, in our framework, the user can encode complex conditions (e.g., a collision) simply by referencing the corresponding constructs in the meaningful event predicates without reimplementing them from scratch (provided that such constructs are present in the simulator, which is one of our assumptions).

We want to clarify that the domain knowledge provided to the system is general to a problem domain and not bound to a particular problem instance. We believe this important point was not efficiently communicated in our manuscript. Specifically, the user provides general principles to search over sequences of meaningful events for an entire problem domain (section 3.2). We have revised our manuscript to clarify this.

Reviewer comment (Theory):
The reviewer asks how the assumption of local searches finding connections between the sequence of "meaningful events" can be guaranteed.

Author response:
We assume that the user has provided events for which this assumption is true. We have revised our manuscript to further highlight this assumption.

In the special case where meaningful events induce convex problems, the connections between events can be guaranteed to be reachable with gradient descent. However, in general, the user is subject to the limitations of the optimization method used to define meaningful events.

Reviewer comment (Experiments):
The fundamental limitation of this approach is that it needs a human to extract what they call "meaningful events" that serves as the foundation of the planning tree. This makes comparisons against model-free methods like RL and sampling-based MPC rather apples-to-oranges. If this step was more automatic, or shown to be very simple, the paper would be much stronger.

Author response:
As you correctly point out, our proposed approach currently relies on a human to provide the space of meaningful events for a given problem domain (and relate them to the source code). Thus, comparing our approach with pure RL and MPC approaches is not completely straightforward because we allow users to provide structured domain knowledge.

We are working on a comparison with planning techniques that leverage temporal specifications. This is more of an apples-to-apples comparison because some of these techniques also allow (and rely) on a human to provide the domain knowledge to make a domain of tasks feasible, and this knowledge is encoded at a comparable level of abstraction.

Finally, we argue that the amount of effort required by our approach is similar to, or smaller than, that required by these techniques. This is because the user can encode complex conditions (e.g., a collision) simply by referencing the corresponding constructs in the meaningful event predicates without reimplementing them from scratch (provided that such constructs are present in the simulator, which is one of our assumptions).

Reviewer comment (Experiments):
This limitation is exacerbated by not using any standard benchmarks that I can see.

Author response:
The reason we did not include the standard RL benchmarks is that our tool requires the code that has to be traced to be Python code. However, in the OpenAI Gym continuous tasks (e.g., half-cheetah, hopper, etc.), the code that encodes the structure of the task is the Mujoco simulator, which is a binary of C source code.

Because of this, we designed the --arguably harder-- "Password Locomotion" task, which leverages the rigid body dynamics as part of a more complex simulation. The simulator makes use of Mujoco but encodes the logic that handles the dynamics of pressing buttons and unlocking the goal in Python.

Reviewer comment (Evaluation):
The success rate curves could also use a confidence interval (or quartiles).

Author response:
These curves are an exact measurement: they are the cumulative number of tasks that are solved over the course of the experiment (which itself consists of multiple tasks). In other words, these curves correspond exactly to the count of successful tasks as the experiment progresses (i.e., they are not an aggregate value), and thus have no confidence interval associated.

Reviewer comment (Experiments):
Why did you not include any standard benchmark environments from e.g. RL as you are comparing against RL?

Author response:
The reason we did not include the standard RL benchmarks is that our tool requires the code that has to be traced to be Python code. However, in the OpenAI Gym continuous tasks (e.g., half-cheetah, hopper, etc.), the code that encodes the structure of the task is the Mujoco simulator, which is a binary of C source code.

Because of this, we designed the --arguably harder-- "Password Locomotion" task, which leverages the rigid body dynamics as part of a more complex simulation. The simulator makes use of Mujoco but encodes the logic that handles the dynamics of pressing buttons and unlocking the goal in Python.