Paper: Addressing Long-Horizon Tasks by Integrating Program Synthesis and State Machines (ICLR-2024)

Reviewer comment (Theory):
The framework utilizes 3 stage design where the learning of the program embeddings and the mode transition function are separated, which could lead to cascading errors.

Author response:
We thank the reviewer for raising this concern. In the following, we discuss how error propagation may happen from Stage 1 (i.e., learning a program embedding space) to Stage 2 (i.e., retrieving mode programs), and from Stage 2 to Stage 3 (i.e., constructing the policy by learning a transition function with the mode programs).

**Stage 1 $\rightarrow$ Stage 2**: Stage 1 learns a program embedding space that continuously parameterizes diverse programs. Once the reconstruction loss is optimized, the program decoder can ideally decode all the training programs given corresponding program embeddings. In fact, the experiments show that with a sufficient embedding size, we can achieve 100% reconstruction accuracy on the entire training program set. Moreover, it can generate novel programs by interpolating in the learned program embedding space, as demonstrated in Trivedi et al., 2021. We believe the cascading error issue would not happen between Stage 1 and Stage 2, i.e., the training programs would not be "dropped/lost" in Stage 1.

**Stage 2 $\rightarrow$ Stage 3**: Stage 2 retrieves a set of programs as modes of a state machine, and Stage 3 learns a transition function based on the retrieved mode programs. Therefore, as pointed out by the reviewer, if none of the programs retrieved from Stage 2 can solve a part of a task, Stage 3 cannot recover from such failure. To maximally avoid this issue of missing programs in Stage 2, this work focuses on designing search methods that can reliably retrieve all the essential programs in Stage 2. Specifically, we propose a CEM-based search method considering a set of programs' effectiveness, diversity, and compatibility. The ablation studies compare various methods for program retrieval and verify the superior performance of our proposed search method.

Reviewer comment (Theory):
The absence of strict constraints to guarantee compatibility between generated programs could result in execution incompatibilities.

Author response:
We thank the reviewer for raising this question. Each decoded program is guaranteed to be executable because a syntax mask is applied to the decoder during the program decoding process proposed by Trivedi et al., 2021 to ensure syntactic correctness. That said, we can safely (i.e., it does not “fail”) execute any programs in any order. 

We would like to clarify that we introduced the compatibility bonus to account for task performance instead of executability. For example, given a set of retrieved programs $p_1$ and $p_2$, the compatibility bonus tends to retrieve the third mode program $p_3$ if executing some orders (e.g., $p_1 \rightarrow p_3 \rightarrow p_2$,  $p_2 \rightarrow p_3$) of these three programs yields satisfactory rewards.

Reviewer comment (Experiments):
The experiments only consider simple tabular tasks, and only a few baselines are compared. Demonstrating the effectiveness of the framework on more complex tasks would strengthen the paper.

Author response:
**Baselines**: We believe that our comparisons to the baselines are sufficient. HPRL (Liu et al., 2023) is the current state-of-the-art method published at ICML 2023 (July 2023). We have revised the paper to include comparisons to the Moore machine network framework proposed by Koul et al., 2019, and state machine policies inspired by Inala et al., 2020. We would appreciate it if the reviewer could be specific about which methods we should compare against.

**Tasks**: We would like to emphasize that the tasks in the Karel domain characterize various properties that resemble hard RL problems, such as long horizons (up to tens of thousands of time steps),  sparse rewards, deceptive rewards, etc. Nevertheless, we agree with the reviewer that extending our work to a variety of domains, e.g., ViZDoom (Kempka et al., 2016), 2D Minecraft (Sun et al., 2020), gym-minigrid (Chevalier-Boisvert et al., 2023), robot arm manipulation (Liang et al., 2023, Wang et al., 2023), is a promising research direction.

Reviewer comment (Experiments):
In Sec. 4.2.3, the most compatible program is selected by inserting the newly generated program into a randomly chosen program sequence and ranking the returns of the execution, would this design guarantee compatibility? If not, what’s the failure ratio of the case that the generated program cannot be executed due to the compatibility issue?

Author response:
We thank the reviewer for raising this question. Each decoded program is guaranteed to be executable because a syntax mask is applied to the decoder during the program decoding process proposed by Trivedi et al., 2021 to ensure syntactic correctness. That said, we can safely (i.e., it does not “fail”) execute any programs in any order. 

We would like to clarify that we introduced the compatibility bonus to account for task performance instead of executability. For example, given a set of retrieved programs $p_1$ and $p_2$, the compatibility bonus tends to retrieve the third mode program $p_3$ if executing some orders (e.g., $p_1 \rightarrow p_3 \rightarrow p_2$,  $p_2 \rightarrow p_3$) of these three programs yields satisfactory rewards.

Reviewer comment (Evaluation):
It seems the method does not perform very ideally in the INF-DOORKEY and INF-HARVESTER, could authors provide additional explanations on this? Including more analysis of the failure modes would also help readers understand the limitations of the work.

Author response:
We thank the reviewer for raising this question. 

**Inf-Harvester**: We noted that in the Inf-Harvester task, there is a drastic difference between retrieving programs and learning the transition function. Specifically, when retrieving mode programs, it seldom starts executing program candidates from a state with a small number of markers, and therefore, the retrieved programs modes mainly are specialized in collecting “dense” markers; however, when learning the transition function, after executing a few programs, the state contains only a small number of markers, and no program mode is specialized in such a scenario. 

**Inf-DoorKey**: Our proposed POMP achieves the second-best performance in the Inf-DoorKey task, with an average reward of 0.91, outperforming HPRL (0.15) and LEAPS (0.1) by a large margin. DRL achieves the best performance (0.97). The Inf-DoorKey environment consists of four small chambers (i.e., one 3x3, two 2x3, and one 2x2) connected to each other. Therefore, each stage of Inf-Doorkey requires only moderate exploration, allowing DRL to achieve good performance.

Reviewer comment (Novelty):
Some related works also explore state machines for high-level transition abstraction in RL, e.g., integrating symbolic planning and skill policies [1][2][3], please consider discussing or comparing.

Author response:
We thank the reviewer for pointing out these relevant works on symbolic planning, and we have revised the paper to discuss them. The major difference between POMP and Cheng and Xu, 2023 and Lin et al., 2022 is the interpretability of the skill or option. In POMP, each learned skill is represented by a human-readable program. On the other hand, neural networks used by Cheng and Xu, 2023 and tabular approaches used by Lin et al., 2022 are used to learn the skill policies. In Yang et al., 2018, the option set is assumed as input without learning and cannot be directly compared with Cheng and Xu, 2023, Lin et al., 2022 and POMP.

Another difference between the proposed POMP framework and Cheng and Xu, 2023, Lin et al., 2022, and Yang et al., 2018 is whether the high-level transition abstraction is provided as input. In Cheng and Xu, 2023, a library of skill operators is taken as input and serves as the basis for skill learning. In Lin et al., 2022, the set of “landmarks” is taken as input to decompose the task into different combinations of subgoals. In PEORL, the option set is taken as input, and each option has a 1-1 mapping with each transition in the high-level planning. On the other hand, the proposed POMP framework utilized the set of retrieved programs as  modes, which is conducted based on the reward from the target task without any guidance from framework input.

Reviewer comment (Novelty):
Reference: [1] LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulation, RA-L 2023; [2] Leveraging approximate symbolic models for reinforcement learning via skill diversity, ICML 2022; [3] PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Making, IJCAI 2018.

Author response:
We thank the reviewer for pointing out these relevant works on symbolic planning, and we have revised the paper to discuss them. The major difference between POMP and Cheng and Xu, 2023 and Lin et al., 2022 is the interpretability of the skill or option. In POMP, each learned skill is represented by a human-readable program. On the other hand, neural networks used by Cheng and Xu, 2023 and tabular approaches used by Lin et al., 2022 are used to learn the skill policies. In Yang et al., 2018, the option set is assumed as input without learning and cannot be directly compared with Cheng and Xu, 2023, Lin et al., 2022 and POMP.

Another difference between the proposed POMP framework and Cheng and Xu, 2023, Lin et al., 2022, and Yang et al., 2018 is whether the high-level transition abstraction is provided as input. In Cheng and Xu, 2023, a library of skill operators is taken as input and serves as the basis for skill learning. In Lin et al., 2022, the set of “landmarks” is taken as input to decompose the task into different combinations of subgoals. In PEORL, the option set is taken as input, and each option has a 1-1 mapping with each transition in the high-level planning. On the other hand, the proposed POMP framework utilized the set of retrieved programs as  modes, which is conducted based on the reward from the target task without any guidance from framework input.

Reviewer comment (Evaluation):
The paper’s main motivation is around interpretability, and it states a separation between FSM and programmatic solutions, where the former allows for repetitive behaviors and the latter for interpretability. However, the claim that FSMs are not interpretable is contested, as the reviewer believes FSMs learned with Inala et al.’s system could also be interpretable. No paper has apparently evaluated the interpretability of these systems, so the argument against FSM interpretability is seen as unfounded.

Author response:
We thank the reviewer for pointing that out. We agree with the reviewer that symbolic programs used in Inala et al., 2020's systems are more interpretable than neural network policies. However, we still believe that such symbolic mode programs describing the relationship among variables, which are presented in Figure 16 to Figure 21 in Inala et al., 2020 paper, are less interpretable than our programs written in a formal language. Trivedi et al., 2021 demonstrated that users with basic programming skills can interpret, edit, and improve the programs synthesized by LEAPS, which highlights the interpretability of DSL programs. 

As suggested by the reviewer, we have revised the paper and toned down our statement. Instead of stating that the FSMs used in Inala et al., 2020 are "difficult to be interpreted by human users," we now argue such FSMs "are less interpretable (compared to DSL programs)."

Reviewer comment (Evaluation):
It is unclear whether the process of learning modes is accounted for in the curves shown in Figure 6a and the numbers in Table 2, and whether learning of modes is performed in the target task. Given the compatible constraint, learning modes could essentially solve the problem by combining programs to maximize expected reward. The paper also lacks clarity on the number of training steps used for each system.

Author response:
We thank the reviewer for raising the questions.
- Yes, timesteps of learning modes are accounted for in the curves shown in Figure 6a. The detail about how returns in Figure 6a are calculated is addressed in Section C of the revised paper.
- Yes, the mode program retrieving process introduced in Section 4.2.3 is performed on each target task to ensure that the retrieved mode programs are diverse and compatible for solving the given task.

Reviewer comment (Experiments):
POMP was not evaluated on the original Karel problems from LEAPS and HPRL, such as the DoorKey problem. The reviewer suspects that POMP might struggle in cases like obtaining the key but not opening the door, and might take longer than LEAPS and HPRL to reach the same reward. The reviewer wants to see POMP results on the original problem set for direct comparison.

Author response:
We have revised the paper to include the performance of DRL, LEAPS, HPRL, and our proposed framework on the Karel problem set and the Karel-Hard problem set in Table 3 (Section G and Section H). We also include the table in this response below to make it easier for the reviewer to read. 

| Method | FourCorner | TopOff | CleanHouse | StairClimber | Harvester | Maze | DoorKey | OneStroke | Seeder | Snake |
|---|---|---|---|---|---|---|---|---|---|---|
| **DRL** | 0.29 $\pm$ 0.05 | 0.32 $\pm$ 0.07 | 0.00 $\pm$ 0.00 | **1.00** $\pm$ 0.00 | 0.90 $\pm$ 0.10 | **1.00** $\pm$ 0.00 | 0.48 $\pm$ 0.03 | **0.89** $\pm$ 0.04 | 0.96 $\pm$ 0.02 | **0.67** $\pm$ 0.17 |
| **LEAPS** | 0.45 $\pm$ 0.40 | 0.81 $\pm$ 0.07 | 0.18 $\pm$ 0.14 | **1.00** $\pm$ 0.00 | 0.45 $\pm$ 0.28 | **1.00** $\pm$ 0.00 | 0.50 $\pm$ 0.00 | 0.65 $\pm$ 0.19 | 0.51 $\pm$ 0.21 | 0.21 $\pm$ 0.15 |
| **HPRL** | **1.00** $\pm$ 0.00 | **1.00** $\pm$ 0.00 | **1.00** $\pm$ 0.00 | **1.00** $\pm$ 0.00 | **1.00** $\pm$ 0.00 | **1.00** $\pm$ 0.00 | 0.50 $\pm$ 0.00 | 0.80 $\pm$ 0.02 | 0.58 $\pm$ 0.07 | 0.28 $\pm$ 0.11 |
| **POMP** | **1.00** $\pm$ 0.00 | **1.00** $\pm$ 0.00 | **1.00** $\pm$ 0.00 | **1.00** $\pm$ 0.00 | **1.00** $\pm$ 0.00 | **1.00** $\pm$ 0.00 | **1.00** $\pm$ 0.00 | 0.62 $\pm$ 0.01 | **0.97** $\pm$ 0.02 | 0.36 $\pm$ 0.02 |

We briefly summarize our findings as follows.
- POMP can fully solve DoorKey by getting the key, opening the door, and placing the key at the desired location. DRL, LEAPS, and HPRL can only obtain the key. Specifically, POMP learns to use the following program to solve the second stage of the DoorKey task, trying every grid to find the correct grid to place the key.
```
DEF run m(WHILE c( leftIsClear c) w( turnRight move putMarker pickMarker move w) m)
```
- POMP outperforms or performs competitively to DRL, LEAPS, and HPRL on 8 out of 10 tasks. 
- When POMP searches for the first mode program, the program retrieval process will be entirely equal to LEAPS, and POMP will take the exact computation cost to reach the same reward as LEAPS. On the other hand, HPRL requires more computational costs than POMP to earn the 1.0 reward on the Karel problem set. For DoorKey and Seeder, HPRL requires more computational costs than POMP to earn the same reward. For OneStroke and Snake, POMP requires more computational costs than HPRL to earn the same reward.

Reviewer comment (Experiments):
The experiments lack a baseline that learns FSMs, such as the method by Koul et al. (Learning Finite State Representations Recurrent Policy Networks), especially since the reviewer was unconvinced by the interpretability argument against FSMs.

Author response:
We thank the reviewer for this suggestion. We agree that including comparisons to methods that produce state machine policies will strengthen our work. Therefore, we have implemented and evaluated a state machine policy, dubbed programmatic state machine policy (PSMP), whose modes are primitive actions of the agent, instead of mode programs retrieved by our method, resembling the idea of Inala et al., 2020. Also, we have implemented and evaluated the Moore machine network (MMN; Koul et al., 2019). The results are reported in Table 2 and discussed in Section 5.4 of the revised paper. We have also included the results below. 

| Method | Seesaw | UP-N-Down | Farmer | Inf-DoorKey | Inf-Harvester |
|---|---|---|---|---|---|
| **PSMP** | 0.25 $\pm$ 0.39 | 0.00 $\pm$ 0.00 | 0.58 $\pm$ 0.06 | **0.97** $\pm$ 0.01 | 0.59 $\pm$ 0.06 |
| **MMN** | 0.10 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 0.57 $\pm$ 0.12 | 0.87 $\pm$ 0.05 | 0.41 $\pm$ 0.10 |
| **POMP** | **0.90** $\pm$ 0.02 | **0.97** $\pm$ 0.00 | **0.88** $\pm$ 0.01 | 0.91$\pm$ 0.01 | **0.67** $\pm$ 0.03 |

As shown above, our proposed POMP consistently outperforms MMN across all the tasks and achieves better performance than PSMP on 4 out of 5 tasks by large margins. More details of these two methods and their performance can be found in the revised paper. We sincerely thank the reviewer for this suggestion, which helps us improve our work.

Reviewer comment (Experiments):
The experiments are missing a baseline that searches directly in the space of programs, as in genetic programming. For example, the method by Aleixo & Lelis (Show Me the Way! Bilevel Search for Synthesizing Programmatic Strategies) uses a two-level search with simulated annealing, and a comparison to this would be useful.

Author response:
We thank the reviewer for suggesting these works. However, Bi-S proposed in Aleixo & Lelis, 2023 and Neurally Directed Program Search (NDPS; Verma et al., 2018) require feature functions as input or hand-crafted program sketches, which differs from our experimental settings that only produce programmatic policies from reward signals. We have revised the paper to include a discussion on Aleixo & Lelis, 2023, as the original paper has referred to Verma et al., 2018.

Reviewer comment (Experiments):
The experiments do not consider NDS by Verma et al. (Programmatically interpretable reinforcement learning), and it is questioned whether it could be used to find programs for Karel.

Author response:
We thank the reviewer for suggesting these works. However, Bi-S proposed in Aleixo & Lelis, 2023 and Neurally Directed Program Search (NDPS; Verma et al., 2018) require feature functions as input or hand-crafted program sketches, which differs from our experimental settings that only produce programmatic policies from reward signals. We have revised the paper to include a discussion on Aleixo & Lelis, 2023, as the original paper has referred to Verma et al., 2018.

Reviewer comment (Reproducibility):
Several key pieces of information are not clearly described in the main text of the paper, which makes it difficult to understand the exact proposed approach and to reproduce the results.

Author response:
We sincerely thank the reviewer for pointing these out. We have thoroughly revised the paper to include the missing details and make the existing information easier to find.

Reviewer comment (Experiments):
How are the programs over which the embedding space is learned generated? Are they simply randomly sampled from the Karel DSL? These programs form such a crucial building block for the method that it's worth discussing how they're generated here, and how they might be generated in other domains.

Author response:
As described in Section 4.1, we follow the training procedure proposed by LEAPS (Trivedi et al., 2021), which generates 50000 programs that are randomly sampled according to the Karel DSL to train the encoder-decoder model. Further details about the program dataset generation and the training of the encoder-decoder model can be found in Section F.1 of the revised paper. 

To generate programs in another domain, we can simply randomly sample programs according to the DSL in that domain "for free" and then learn a program embedding space.

Reviewer comment (Reproducibility):
How are the rewards defined in the CEM optimization in the retrieval stage? It seems like one would need many different reward functions to ensure good diversity and coverage in the selected modes. Is it task-specific? This crucial information does not appear to be clearly explained anywhere.

Author response:
The reward used by POMP for CEM optimization, as shown in Equation 1 and Figure 3, is defined as: 

$R\_{\rho} = \sum\_{i=1}^{|\Psi\_{\text{before}}|} \gamma ^ {i-1} \sum\_{t=0}^{T^i} \gamma^t \mathbb{E}\_{(s\_t,a\_t) \sim \text{EXEC} (\Psi\_{\text{before}}[i])}[r\_{t}] + \gamma^{|\Psi\_{\text{before}}|}\sum\_{t=0}^{T} \gamma^t \mathbb{E}\_{(s\_t,a\_t) \sim \text{EXEC} (\rho\_z)}[r\_{t}]  + \gamma^{|\Psi\_{\text{before}}|+1}\sum\_{i=1}^{|\Psi\_{\text{after}}|} \gamma^{i-1} \sum\_{t=0}^{T^i} \gamma^t \mathbb{E}\_{(s\_t,a\_t) \sim \text{EXEC} (\Psi\_{\text{after}}[i])}[r\_{t}].$

Note that this reward function is not task-specific, and we use this reward to retrieve the set of mode programs for POMP across all tasks discussed in Section 5.1.

Reviewer comment (Reproducibility):
How is the transition function f learned? Is this learned via standard RL, where choosing the modes is the action?

Author response:
As stated in Section 4.3 and Section F, the learning process of transition function $f$ is formulated as a standard RL problem. A state includes the current Karel grid and the previously executed mode represented by a one-hot vector, while an action chooses the next mode program, as correctly understood by the reviewer. We use the PPO algorithm to optimize the transition function $f$.

Reviewer comment (Reproducibility):
Details about the state are missing from the main body of the work, which makes it difficult to understand the task. From what I gather from the Appendix, a N x M array is used as the state for a N x M grid world for the deep RL baseline — is this the same state that the proposed approach uses for selecting which mode to switch to?

Author response:
We thank the reviewer for bringing up this issue. Yes, as described in Section 4.3 and Section F of the revised paper, both the DRL baseline and the mode transition function of POMP use a CNN to encode an input state of an N x M array, describing the current Karel grid, as shown in Figure 5 and Section H of the revised paper.

Reviewer comment (Experiments):
The deep RL baseline seems to be surprisingly weak, given that the task is just grid world pick and place. Is there a reason behind this? Is it a result of the state representation? Does e.g., using a different state representation of just the agent's (x, y) coordinates work better?

Author response:
We thank the reviewer for mentioning this observation. DRL struggles at the tasks in the Karel-Long problem set that require long-horizon exploration with sparse rewards (i.e., Seesaw and Up-N-Down), as described in Section I. On the other hand, DRL can achieve satisfactory performance on the tasks which provide dense rewards (i.e., Farmer, Inf-DoorKey, and Inf-Harvester), even when the episode horizon is long.

Reviewer comment (Experiments):
While the details for how the entire space of programs is generated are missing, it seems difficult to imagine being able to do this well for arbitrary domains. In general, writing programs that can easily compose is challenging, particularly when they have inputs and outputs, whereas this domain is particularly simple and the programs do not need to take arguments or return values. Consequently, there is concern that the proposed approach may not be able to be applied to other less toy domains. Discussion about this would be very helpful.

Author response:
We thank the reviewer for raising this concern. We certainly agree with the reviewer that the proposed framework, and most programmatic reinforcement learning methods, are not generally applicable to arbitrary domains.

Our program policies are designed to describe high-level task-solving procedures or decision-making logics of an agent. Therefore, our principle of designing domain-specific languages (DSLs) considers a general setting where an agent can perceive and interact with the environment to fulfill some tasks. DSLs consist of control flows, perceptions, and actions. While control flows are domain-independent, perceptions and actions can be designed based on the domain of interest, which would require specific expertise and domain knowledge. 

Such DSLs are proposed and utilized in various domains, including ViZDoom (Kempka et al., 2016), 2D Minecraft (Sun et al., 2020), and gym-minigrid (Chevalier-Boisvert et al., 2023). Recent works (Liang et al., 2023, Wang et al., 2023) also explore describing agents’ behaviors using programs with functions taking arguments. We have revised the paper to include this discussion (Section J).

Reviewer comment (Theory):
It also seems generally difficult to span a set of useful behaviors needed in the programs. How can this be achieved without ending up with an intractable space of programs to search over?

Author response:
Following LEAPS (Trivedi et al., 2021) and HPRL (Liu et al., 2023), we construct a program embedding space that continuously parameterizes diverse programs by randomly generating a program dataset. We limit the program length when generating the dataset, which subsequently confines the possible program space not to be unlimitedly large or intractable, and therefore the dataset programs describe shorter-horizon behaviors.

As pointed out by the reviewers, the majority of the generated programs do not capture meaningful behaviors. Therefore, devising efficient search methods that can quickly identify effective programs is critical, which is the main focus of LEAPS and HRPL. The search method proposed in our work, on the other hand, not only aims to obtain effective programs but is also designed to retrieve a set of effective, diverse, and compatible programs. For example, Figure 7 illustrates how the proposed CEM+diversity can avoid searching over paths similar to those previously explored, which drastically increases the searching efficiency.