Paper: Reclaiming the Source of Programmatic Policies: Programmatic versus Latent Spaces (ICLR-2024)

Reviewer comment (Reproducibility):
What’s the maximum length of the sampled programs?

Author response:
**Question:** What’s the maximum length of the sampled programs?

**Answer:** 45 symbols in the program's text representation.

Reviewer comment (Experiments):
What is the exact height of the abstract syntax tree (AST) of every program?

Author response:
**Question:** What is the exact height of the abstract syntax tree (AST) of every program?

**Answer:** The maximum height of each program is 4.

Reviewer comment (Reproducibility):
The above settings should be “similar” to LEAPS based on the description of Section 3.1, but are they identical to all the settings used in LEAPS in all experiments? What are the actual numbers used for the search in the programmatic space?

Author response:
**Question:** The above settings should be “similar” to LEAPS based on the description of Section 3.1, but are they identical to all the settings used in LEAPS in all experiments? 

**Answer:** Yes, they are identical to all the settings used in LEAPS in all experiments. We clarify this in a sentence in Section 3.1 of the revised version and in the Appendix A. 

**Question:** What are the actual numbers used for the search in the programmatic space?

**Answer:** Please see our answers above.

Reviewer comment (Novelty):
Overall, I do not find enough novelty in this work but the overall effort of this paper is appreciated.

Author response:
**Comments on Novelty**

We agree that our work does not introduce a new method and, in this regard, is not novel. Our contribution is unusual compared to most papers, as we aim to influence the research direction of the sub-field by providing a baseline missed in previous work. The results of this baseline raise fundamental questions related to how the community is using latent spaces in the context of programmatic policies. Although the contribution is unusual, we don't see it being less important than published papers proposing novel algorithms.

Reviewer comment (Reproducibility):
Because the local search in the programmatic space is not continuous, the initial candidate and the randomness of the programs sampled from the distribution will dominate the quality of the search results. However, there is no further detail about how the initial candidate is sampled and how many random seeds are used to evaluate the search result. Such ambiguity makes it hard to assess the robustness and efficacy of this work.

Author response:
**Re: Distribution of Programs.** We use the distribution of programs from the LEAPS paper to sample initial programs and to sample neighbors of a candidate program. We added a section in the Appendix describing the process used in the LEAPS work, which we adopted in our work (see page 12 of the revised version). 

**Re: Number of Seeds.** In our original submission we didn't use the word "seeds" to denote independent runs of the system, and that explains why the reviewer might have missed this information in the paper. In the revised version, we updated the text to always use the word "seeds". The smallest number of seeds used was 32.

Reviewer comment (Experiments):
The paper makes vague promises that are either not concrete or not trivially feasible to me. For example, the authors state in Section 5.2 that if the search algorithm fails to converge but its execution is still within the budget, an initial program is re-sampled and restart the search. I am not sure how often this kind of failure would happen. Since the budget is high (10^6) in this work, the search will be like a brute-force searching paradigm if the failure rate is high.

Author response:
**Re: Brute-Force Search.** The size of the space is so large that $10^6$ is far from approaching the size of the space. The plots in Figure 5 provide the information on how often the restarts happen. Since the plots show the convergence rate $p$, the failure rate is $1 - p$. Note that restarting the search is standard in local search algorithms.

Reviewer comment (Experiments):
Why use fixed random seeds for the experiments as described in Section 3.1? Will the program search in the programmatic space work properly under different initial seeds? Will HC converge in all tasks in Table 1 under different random seeds?

Author response:
**Re: Fixing Seed.** We only fix the seed to be able to talk about an optimization landscape. Fixing the seed is standard in any ML experiment: every time we run an experiment, we fix the seed and run the experiment; this process is repeated and results are averaged. Your comment made us realize that our wording was confusing. Since fixing seeds is standard practice in all experiments in the field, we removed that sentence from the paper.

Reviewer comment (Reproducibility):
In the Hill Climbing search, how do the authors choose the best-seen candidate if all candidates have the same episodic return (e.g., 0.0)?

Author response:
**Re: All Candidates with Same Return.** We break ties arbitrarily in all search algorithms. That is, if two candidate solutions have the same value (e.g., 0.0), then we arbitrarily select one of them.  We added a sentence in the first paragraph of Section 4 of the revised version explaining this.

Reviewer comment (Evaluation):
As described in the second paragraph of Section 5.2, If an algorithm fails to converge but its execution is still within the budget, an initial program is re-sampled, and restart the search. What is the failure rate of HC and CEBS in each task in Table 1?

Author response:
**Re: Failure Rate.** The failure rate can be computed from Figure 5, as it is given by $1 - p$, where $p$ is the convergence rate and provided in Figure 5. Note that Figure 5 provides $p$ for HC for both spaces and not for CEBS. This is because we are interested in the property of the space, which HC is able to provide because it terminates once it encounters a local optimum. We provide the convergence rate of CEM and CEBS in Appendix I, for completeness.

Reviewer comment (Reproducibility):
How many seeds are used for each algorithm in Table 1 and Figure 3,4,5?

Author response:
**Re: Number of Seeds.** As explained above, we changed the text to explicitly talk about "seeds". See the revised version of the paper. For Table 1 we used 32 seeds, for Figures 3, 4, and 5 we used 32, 1,000, and 10,000 seeds, respectively.

Reviewer comment (Reproducibility):
The details of the initial candidate programs are missing. How is the initial candidate determined to construct the search graph for each task? How long is the initial candidate program? From what probability distribution is it sampled?

Author response:
**Re: Information About Candidates.** We added this information in the Appendix of the revised version of the paper, where we explain the process used in the LEAPS paper that we adopted in our experiments.

Reviewer comment (Reproducibility):
The implementation details are lacking from the main paper, which makes reproducing the results difficult.

Author response:
**Reproducibility**

We will make our code available after the review process, so the results can be easily reproduced.

Reviewer comment (Reproducibility):
No figure or pseudocode of the Hill Climbing and the Cross Entropy Beam Search is provided for the Karel environment, which makes it hard to testify and evaluate the effectiveness of the proposed method.

Author response:
**Re: No Pseudocode.** We added the pseudocode for Hill Climbing and CEBS to the Appendix; see pages 12-13 of the revised version.

Reviewer comment (Evaluation):
No program from the search in programmatic space is shown in the paper, making it hard to assess the efficacy of program search in the programmatic space.

Author response:
**Re: Example Programs.** We added one program for each problem domain in the Appendix; see pages 15-16 of the revised version.

Reviewer comment (Evaluation):
The experiment about convergence analysis is not convincing and could be misleading. In Table 1, the HC can achieve a score of 0.84 on task "DoorKey" while the corresponding convergence rate is 0.0 beyond the episodic return of 0.5 in Figure 5 (A similar observation can also be found on task "Snake"). Is it a contradiction, or the result of HC in Table 1 is based on rare cases?

Author response:
**Experimental Conclusions**

Thank you for pointing this out. The discrepancy in the results are due to very rare events. The convergence rate experiments in Figure 5 were performed with 250 seeds, while the search budget for Table 1 and Figure 3 is 1,000,000. Rare events from the results in Table 1 and Figure 3 may not be observed with only the 250 seeds used in Figure 5. We replaced the plots in Figure 5 with plots generated with 10,000 seeds. The plots for DoorKey and Snake offer a zoomed-in highlight of the rare events. We also added a sentence in the main text explaining this discrepancy (see Section 5.3.2 of the revised version).

Reviewer comment (Experiments):
The experiment enviornment is a rather simple one and use of knowledge of the simple grid world environment reduces strength of the proposed method.

Author response:
We did not choose the problem domain nor the domain-specific language. We are comparing a simple method using the same methodology used in recent previous work. Having said that, the methodology used in previous works is fine. Karel is a challenging domain and helpful for driving research in reinforcement learning.

Reviewer comment (Evaluation):
Performance mismatch: Table 1 and Figure 5 please check.

Author response:
Could you please explain what the mismatch is? Perhaps you are referring to the convergence result providing a value of zero for high reward values for DoorKey and Snake, as reviewer BAZd pointed out. The discrepancy in the results are due to very rare events. The convergence rate experiments in Figure 5 were performed with 250 seeds, while the search budget for Table 1 and Figure 3 is 1,000,000. Rare events from the results in Table 1 and Figure 3 may not be observed with the 250 seeds used in Figure 5. We replaced the plots in Figure 5 with plots generated with 10,000 seeds. The plots for DooKey and Snake now offer a zoomed-in highlight of the rare events. We also added a sentence in the main text explaining this discrepancy (see Section 5.3.2 of the revised version).

Reviewer comment (Experiments):
Potential dependance of the performance of the algorithm on initial condidtions. Please test.

Author response:
What do you mean by "initial conditions"? Do you mean the program that is used to seed the search? We generate initial programs using the same distribution of programs used to train the latent spaces in LEAPS and HPRL. We agree that the initial program can influence the search, but testing this is beyond the goal of our study, which is to evaluate latent and programmatic spaces. Since both latent and programmatic spaces use the same distribution of programs, the comparison is fair.

Reviewer comment (Theory):
Please justify why the proposed method "is able to escape such local maxima" and "this is a property of the search space itself”. To this reviewer, it is not straightforward. Either theoretial or numerical evidences should be provided.

Author response:
**Answer:** The search algorithm we use in the programmatic space, Hill Climbing (HC), does not escape local optima because once it finds one, it terminates the search. The only explanation for the HC search in the programmatic space being able to find good solutions to the DoorKey problem must be a property of the search space. 

**Answer:** We provide plenty of empirical evidence for such a claim. Figure 3 shows that HC is able to achieve average rewards higher than 0.5, the local optima the search in the latent space cannot overcome. Figure 5 of the revised version shows the convergence rate for DoorKey, showing that in a small number of runs, HC is able to obtain rewards larger than 0.5.