Paper: LILO: Learning Interpretable Libraries by Compressing and Documenting Code (ICLR-2024)

Reviewer comment (Evaluation):
Evaluation - I am not sure that it is fair to compare the proposed approach to a *prompted* LLM as a baseline when this baseline is prompted with the lambda-calculus language, which it probably was not trained on. Since the task is defined in a domain-specific-language (DSL), a prompted LLM is expected to fail. Can the task be "translated" to Python? If the LLM would be allowed to use Python, that would be a more fair comparison. Alternatively, a more fair baseline could be fine-tuning an LLM on the language, rather than just prompting, since the proposed approach also does kind-of training within its compression step.

Author response:
**Prompting a LLM with a programming language it was not trained on is a bit unfair.**

We use the same few-shot prompting setup for LLM-guided synthesis across all models, including LILO; therefore, any code comprehension issues that affect the LLM baselines will also affect LILO in the same ways. Conversely, if finetuning a LLM on one of our DSLs were to raise the LLM Solver baseline, it would also necessarily improve LILO’s performance. For this reason, we don’t agree with the assessment that there is something “unfair” about the design decision to use few-shot prompting instead of finetuning.

We also want to address the misperception that these DSLs are inherently problematic for a 175B parameter LLM like `code-davinci-002` as this has not been our experience. The DSLs in this work all use LISP syntax, which is prevalent across many modern languages (Scheme / Racket / Clojure / etc.) and are therefore well-represented in code pre-training datasets like GitHub. The main aspect the LLM will be unfamiliar with is the semantics of the DSL, which is precisely the point — our goal is to build a model that can generalize to novel domains based on only a handful of examples and without expensive finetuning.

**Can the tasks be translated into Python?**

Yes—and this is something that we considered early in the project. In practice, however, we found that syntax errors with Codex were rare (e.g., virtually no issues with parenthesis matching). Given that the LLM Solver already showed strong performance on the online synthesis experiments when working in the original syntax, we opted not to introduce additional system complexity of translating the programs into a Python-like syntax. Irrespective of the syntax, translation would not address the fact that the main difficulty for LLM-guided synthesis arises from the novel semantics of the domains and the novel abstractions introduced by compression. Accordingly, our emphasis in this work is on methods to address issues with semantic comprehension (i.e., via AutoDoc) and not syntactic comprehension.

Reviewer comment (Experiments):
Can the task be "translated" to Python, and then tested on an LLM?

Author response:
**Can the tasks be translated into Python?**

Yes—and this is something that we considered early in the project. In practice, however, we found that syntax errors with Codex were rare (e.g., virtually no issues with parenthesis matching). Given that the LLM Solver already showed strong performance on the online synthesis experiments when working in the original syntax, we opted not to introduce additional system complexity of translating the programs into a Python-like syntax. Irrespective of the syntax, translation would not address the fact that the main difficulty for LLM-guided synthesis arises from the novel semantics of the domains and the novel abstractions introduced by compression. Accordingly, our emphasis in this work is on methods to address issues with semantic comprehension (i.e., via AutoDoc) and not syntactic comprehension.

Reviewer comment (Experiments):
Can an LLM such as LLaMA be finetuned on the training examples, to better understand their language? Prompting an LLM with a language it was not trained on is a bit unfair.

Author response:
**Can a LLM (e.g., LLaMA) be finetuned on the DSL?**

We want to emphasize that in our “lifelong learning” evaluation setting, which follows from DreamCoder, **the number of program examples available at the start of learning is very small**—about 10-30 examples, which is likely not enough for finetuning a LLM. Moreover, the goal of this data scarcity is to test models’ ability to perform humanlike generalization from a small handful of examples. Under this view, the pretrained LLM in LILO plays the role of a broad prior that captures some of the common knowledge that a human programmer might bring to a novel problem domain. While it is possible to collect a larger set of annotated examples to finetune a LLM, we believe this is a different kind of evaluation setting that is both less novel and also diverges in motivation from the goals of this line of work.

**Prompting a LLM with a programming language it was not trained on is a bit unfair.**

We use the same few-shot prompting setup for LLM-guided synthesis across all models, including LILO; therefore, any code comprehension issues that affect the LLM baselines will also affect LILO in the same ways. Conversely, if finetuning a LLM on one of our DSLs were to raise the LLM Solver baseline, it would also necessarily improve LILO’s performance. For this reason, we don’t agree with the assessment that there is something “unfair” about the design decision to use few-shot prompting instead of finetuning.

We also want to address the misperception that these DSLs are inherently problematic for a 175B parameter LLM like `code-davinci-002` as this has not been our experience. The DSLs in this work all use LISP syntax, which is prevalent across many modern languages (Scheme / Racket / Clojure / etc.) and are therefore well-represented in code pre-training datasets like GitHub. The main aspect the LLM will be unfamiliar with is the semantics of the DSL, which is precisely the point — our goal is to build a model that can generalize to novel domains based on only a handful of examples and without expensive finetuning.

Reviewer comment (Novelty):
Regarding the AutoDoc module - more than *having* an auto-documentation module, the interesting part to me is that this generated documentation improves the next iteration, when fed to the LLM. That is, the self-improvement part (the fact that the LLM generated documentation improves the LLM itself in the next iteration) is the interesting aspect from an ML perspective to me, while the paper mostly emphasizes the "existence" of this module to generate documentation, which is not significantly novel or interesting on its own.

Author response:
**Continual self-improvement via auto-documentation:** Our results demonstrate that LILO is able to bootstrap LLM synthesis performance via auto-documentation. We are glad that you found this result “interesting from an ML perspective” and agree that this is one of the most exciting selling points of this work. We certainly intended to emphasize the self-improvement aspect of AutoDoc throughout the paper; e.g.,
    - “In addition to improving human readability, we find that AutoDoc boosts performance by helping LILO’s synthesizer to interpret and deploy learned abstractions.” (p. 1, abstract)
    - “yielding better interpretability and improving downstream LLM-guided search” (p. 1)
    - “ Key to this neurosymbolic integration is our AutoDoc module, which not only improves interpretability, but also helps the LLM synthesizer use the library more effectively” (p. 2)
    - “Unlike traditional program synthesis methods, language models draw inferences about the semantics of programs from their lexical content… we explore how AutoDoc benefits downstream synthesis performance” (p. 5)
    - “Auto-documentation unlocks effective contextual usage of abstractions.” (p. 8)

All of the above are intended to emphasize the functional benefits of the AutoDoc module beyond simply pointing to its existence in the architecture. Nevertheless, we certainly want to make sure this point is coming through. We will take your comment under advisement and look for ways to further emphasize this novel aspect in our revised submission.

Reviewer comment (Evaluation):
As the author also mentioned, the pipeline needs an additional self-verification step for the auto-documentation. The validity and conciseness of the language-guided program synthesis is enforced by input-output pairs and the stitch compression, while the library auto-doc has no verification or feedback loop to update the LLM or other trainable part of the pipeline.

Author response:
**As the author also mentioned, the pipeline needs an additional self-verification step for the auto-documentation.**

We generally agree, though we feel that “needs” is a bit of a strong characterization given that we show AutoDoc already helps with performance and is usually correct. However, this is a natural area for follow-up work. While this is not something that we expect to be able to implement during the review response period, it is easy to envision how such a “self-verificaiton” module might be architected following standard LLM prompting patterns.

Additionally, we’ve started working on a small experiment comparing AutoDoc with expert-generated documentation in response to comments from Reviewer oPbs. This may be of interest as it touches on the same question you’ve raised here around whether stronger verification for documentation can improve performance.

Reviewer comment (Experiments):
The application of the method is limited in functional programming with relatively simple DSL. Though it's almost a common limitation of program synthesis models/systems.

Author response:
**The application of the method is limited in functional programming with relatively simple DSL.**

As noted, these are common limitations of program synthesis research. There are good reasons why a lot of work from that community has historically focused on these settings. Functional languages have many properties that make them more amenable to research. For instance, the absence of side-effects are one language feature that make it possible to implement a program compression algorithm like Stitch. Currently, there is no equivalent in imperative languages like Python, though as we note in the last paragraph of Section 5, we view this as a promising and tractable area for future work in PL. Given an appropriate compression algorithm implementation, it would be straightforward to implement LILO for Python!

[1] Chaudhuri, S., Ellis, K., Polozov, O., Singh, R., Solar-Lezama, A., & Yue, Y. (2021). Neurosymbolic programming. *Foundations and Trends® in Programming Languages*, *7*(3), 158-243.

[2] Willard, B. T., & Louf, R. (2023). *Efficient Guided Generation for Large Language Models* (arXiv:2307.09702). arXiv. http://arxiv.org/abs/2307.09702

Reviewer comment (Reproducibility):
What does PCFG stand for? Could the author describe the input-output, architecture, and training of this multi-layer perceptron? Is this model pre-trained? DSL-specific?

Author response:
**What does PCFG stand for?**

A probabilistic context free grammar (PCFG) is a generalization of a context-free grammar that assigns a real-valued weight to each production rule. **We define this term towards the start of the Preliminaries section on p. 3**: “the prior is defined under a probabilistic context free grammar (PCFG; Johnson 1998)...”

More formally, a PCFG can be defined as a quintuple $G = (M, T, R, S, P)$ where $M$ is a set of non-terminal symbols, $T$ is a set of terminal symbols, $R$ is a set of production rules, $S$ is the start symbol, and $P$ is a function that assigns a probability to each rule in $R$. This probabilistic element allows for the generation of a diverse range of structures. Moreover, as we describe below, training a task-conditioned neural network to infer $P$ is a natural way to guide search in the PCFG.

**Could the author describe the input-output, architecture, and training of this multi-layer perceptron? Is this model pre-trained? DSL-specific?**

Happy to provide more clarification! Here, we are talking about a piece of DreamCoder’s enumerative search algorithm that trains a **“recognition network”** (i.e., an RNN encoder + a MLP) to guide enumerative search by inferring $P$: the probabilities on production rules in the PCFG, as defined above.

***All of this machinery is part of prior work (i.e., DreamCoder)*** and uses off-the-shelf Pytorch components, so we don’t allocate much space in the main paper body to describe these details. Nevertheless, these are natural questions for a reader interested in implementation, so we provide an overview of the architecture below. **We refer to Appendix I in the [DreamCoder supplement](https://dl.acm.org/doi/10.1145/3453483.3454080) for further details details and will add a pointer to this appendix in the paper.**

- **Inputs and outputs:** The recognition network takes as input a set of input/output examples for a specific task and produces an $|\mathcal{L}| \times |\mathcal{L}|$ tensor with values in $[0, 1]$ specifying transition probabilities for every production in the PCFG.
- **Architecture/training:** The input/output examples are encoded using an RNN (specifically, a bidirectional GRU) for text-based domains and a CNN for image-based domains. The actual MLP module is very small (only two layers, with a hidden size of 64) so the whole architecture can be learned efficiently in-the-loop. During each learning iteration, the recognition network is trained for a maximum of 10K gradient steps.
- **Pre-training:** As implemented, the model doesn’t use any pre-trained components. Following prior work with these datasets, we assume ground truth access to the CLEVR scene graphs, which allows us to use a text-based encoder for that domain. If we wanted to train on the original images, one could easily swap in a pre-trained CNN, or even a SOTA segmentation model, for the encoder.
- **DSL-specificity:** Overall, the recognition network architecture is fairly general—the main domain-specific pieces are the choice of encoder and the dimensionality of the output, which depends on $|\mathcal{L}|$.

Reviewer comment (Evaluation):
In the Table 1, only in LOGO, the LILO method significantly outperforms the LLM solver + search. Could the author explain the performance difference among the different DSLs? If the reason is that LLM is not pre-trained on only LOGO, it could be an interesting and promising observation about the generalization capability of both LLM and LILO.

Author response:
**Only in LOGO does LILO significantly outperform LLM solver + Search. Could the author explain the performance difference among the different DSLs?**

First, to clarify, this is only the case for online synthesis—we note that **none of the results from offline synthesis (Table 1, bottom half) fall within $1\sigma$ of the best (bolded) result**. These include the two new baselines added during the response period. We also note that LILO’s $L_f$ consistently finds more solutions *faster* than the other libraries evaluated in offline synthesis.

To address your question about LOGO: we find that LOGO is intrinsically more difficult than the other two DSLs across the board, for all models. We attribute this to the fact that many LOGO tasks require discovering domain-specific program structures (e.g., how to draw a “snowflake” or “staircase”; see Fig. 5) that require mastery of concepts like radial symmetry. Anecdotally, the authors also find LOGO to be the hardest DSL to write programs in!

Reviewer comment (Evaluation):
In Table 1, the proposed 'LILO' framework has substantially higher standard deviation than the baseline across three domains. It indicates LILO's performance varies more widely across different runs, suggesting less stability in this field.

Author response:
**In Table 1, the proposed 'LILO' framework has substantially higher standard deviation than the baseline across three domains. It indicates LILO's performance varies more widely across different runs, suggesting less stability in this field.**

Part of the novelty of our method is in the integration with a LLM, which inherently introduces variability across runs. In response to your comments and those of the other reviewers, we have updated Table 1 to more transparently present this variability by underlining all results that fall within $1\sigma$ of the best (bolded) result.

While some of the online synthesis conditions approach LILO’s performance, we note that none of the results from offline synthesis (Table 1, bottom half) fall within $1\sigma$ of the best (bolded) result. Thus, **we are confident that our results indicate that LILO meaningfully outperforms the other models considered in offline synthesis.**

We also would like to highlight that $\sigma$ is computed with respect to relatively small $N=3$ runs due to the computational cost of each full run of online program synthesis, so it is not a very tight approximation of the variability in performance. Prior work (e.g., DreamCoder, LAPS) does not  report this variability as part of the main results, but we felt it important to include some kind of variability estimate in our Table 1.

Reviewer comment (Experiments):
The dual-system search module combines LLM-guided and enumerative search, yet the tradeoff and relative contributions are not deeply analyzed.

Author response:
**The dual-system search module combines LLM-guided and enumerative search, yet the tradeoff and relative contributions are not deeply analyzed.**

Thank you for the suggestion. We have added two new baselines to the “offline synthesis” section for the LLM Solver and LLM Solver (+ Search). While these are primarily intended to address a specific concern from Reviewer oPbs, these baselines also help to further pinpoint the relative contributions of LLM-guided search, enumerative search, and compression within LILO.

Beyond these new baselines, our existing results set already gives significant attention to the question of the relative contributions of the two search methods. As a recap, we include the following baselines/ablations as part of our original experiments:

- LILO (No Search)
- LLM Solver (+ Search)

This pair of conditions (removing enumerative search from LILO / adding enumerative search to the LLM-only baseline) is directly targeted at analyzing the tradeoffs and relative contributions of the two search methods. We discuss these findings on p. 7 (”To isolate the effects of search…”). In addition to the quantitative results, we also mention several qualitative examples of tasks that are solved with enumerative search but not LLM-guided search (e.g., how to draw a “snowflake” or “staircase” in the LOGO domain; see Fig. 5).

Reviewer comment (Experiments):
The proposed 'LILO' is strongly based on LLM and it mentions using different models like Codex, gpt-3.5-turbo, gpt-4, but did not provide direct comparison of the their performance.

Author response:
**The proposed 'LILO' is strongly based on LLM and it mentions using different models like Codex, gpt-3.5-turbo, gpt-4, but did not provide direct comparison of the their performance.**

While we would have liked to be in a position to systematically benchmark various LLMs for LLM-guided synthesis, as we discuss in Footnote 1, cost considerations required us to restrict our analysis to Codex.

As a concrete example, based on our numbers in **Appendix C.1, Table 6**:

- An equivalent benchmark of **gpt-3.5-turbo** would have cost **$1,272.36** at the current price of $0.002 / token.
- An equivalent benchmark of **gpt-4** would have cost **$38,170.80** at the current price of $0.06 / token.

In contrast, we accessed Codex through OpenAI’s free beta program for researchers, which saved thousands of USD over the project lifetime and afforded higher rate limits than paid GPT models.

As discussed in implementation details, we were able to make use of gpt-3.5-turbo and gpt-4 for AutoDoc, which requires orders of magnitude fewer queries. This is why these models are mentioned in the paper, but we make no claims about having systematically benchmarked these other models for synthesis. We also do not think that exhaustive and costly benchmarking of an ever-changing set of foundation models should be a standard for publication at a conference such as ICLR.

Reviewer comment (Reproducibility):
In Figure 10 caption D, the authors mentions LLM output are sampled without clearly specify the filtering/selection standard. I advise the author can give a more detailed explanation.

Author response:
**In Figure 10 caption D, the authors mentions LLM output are sampled without clearly specify the filtering/selection standard. I advise the author can give a more detailed explanation.**

Thank you for the attention to detail. This filtering/selection pipeline is already described in Section 3 ("For each completion, we run parsing, type inference, and execution checks to identify valid programs that solve the target task.”). To provide more detail, our post-processing pipeline for LLM completions contains the following steps:

Given a `completion: str`

1. Check whether `completion` parses to a valid program in the DSL. If not, discard.
2. Check whether we can perform type inference on `completion`. If not, discard. At this point, `completion` represents a valid, well-typed `program` in the DSL.
3. Check whether `program` satisfies the target task specification (i.e., execute it against all I/O examples). If it does, add `program` to the set of task solutions.

If there are any specific aspects that you would like more detail on, please let us know. Our code on GitHub (currently anonymized for review) will also provide a clear reference for how this pipeline is implemented.

Reviewer comment (Experiments):
To better separate how the contributions of LILO (LLM program induction in the context of iterative library learning, along with auto-documentation of abstractions) affect system performance, the paper should consider Condition (A): Base DSL + STITCH. This would be an ablated version of LILO, where no LLM calls are made. “Wake” would be performed through enumerative search, and STITCH would still be used to generate new libraries. From the reviewer’s perspective, this ablation condition is critical to supporting the claims of the paper; at present, it is possible that the delta improvement between LILO and DreamCoder is due entirely to LILO’s use of STITCH, and ablating the use of LLMs in this way would provide a definitive answer.

Author response:
We appreciate your attention to detail around the experiment design and hope to fully address your concerns below. Concretely, the following two experimental conditions were proposed.

## Baseline A: Base DSL + Stitch

Proposal: This baseline is similar to a DreamCoder + Stitch integration, where Stitch replaces DreamCoder’s compression module. The motivation for this condition is to address the possibility of a confound due to the choice of compression module, which could account for the performance delta between LILO and DreamCoder in the experiments.

*Response:* Your concerns about this confound highlight an oversight in our methods presentation: **Stitch is already used as the compression module for all models reported in the paper. This includes our DreamCoder condition, which corresponds exactly to the proposed “Baseline A.”** Specifically, we swapped out DreamCoder’s compression module to use Stitch while preserving the rest of the architecture. There were two motivations for this change:

(1) To improve the efficiency of the “sleep” phase: for even modestly-sized datasets (i.e., hundreds of programs), Stitch makes the difference between compression running in tens of seconds vs. tens of hours (see the performance benchmarks in [1]; we find similarly striking efficiency improvements on our domains).

(2) To facilitate direct comparability between LILO and DreamCoder and remove the possibility of the confound described here.

**Thank you for highlighting for this oversight—this is precisely the kind of background assumption that would be difficult for the authors to catch in our own work. We promise to clarify this point in our updated submission!**

Reviewer comment (Experiments):
The paper should also consider Condition (B): LLM Solver+Search+STITCH. This condition would first employ the LLM Solver+Search for a set amount of iterations, and then would run STITCH over the final set of inferred programs to output a new library. This would be important to support the implicit claim that library learning needs to be performed in the intermediate stages of the algorithm. If LILO outperforms this ablation on offline search performance (e.g., bottom half of Table 1), then it would be strong evidence in favor of the method.

Author response:
## Baseline B: LLM Solver + Search + Stitch

Proposal: This baseline is an extension of the existing LLM Solver (+ Search) condition, where Stitch is run on the final iteration. The motivation for this condition is to produce a library that can be evaluated in offline synthesis so as to isolate the effect of intermediate compression on performance. The reviewer states that if LILO were to outperform this ablation in offline synthesis, it would be a “strong mark in favor of the method.”

****Response:**** We immediately saw the value of this proposal and ran this baseline over the weekend. For completeness/consistency with the online synthesis conditions, we also ran the analogous version of this baseline for the LLM Solver (no Search) condition, which we expect to be weaker than LLM Solver (+ Search).

After running the experiment, we can now confidently report that LILO outperforms both of these baselines by a good margin ($>1\sigma$) on all domains. As intended, the only implementational difference between LILO and the new LLM Solver (+ Search) baseline with post-hoc Stitch compression is the intermediate compression + documentation steps. Therefore, we agree that this result makes a strong argument in favor of performing library learning in-the-loop.

|  |  | REGEX |  |  | CLEVR |  |  | LOGO |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | max | mean | std | max | mean | std | max | mean | std |
| LLM Solver | 48.60 | 43.00 | 5.17 | 91.26 | 89.64 | 2.02 | 36.04 | 27.33 | 7.56 |
| LLM Solver (+ Search) | 63.40 | 55.67 | 7.51 | 91.26 | 89.00 | 3.92 | 28.83 | 27.63 | 1.04 |

**We have updated our submission with these new results (Table 1 and Figure 4)—please see the updated PDF!** Thank you for this insightful suggestion—we certainly feel these new baselines strengthen the empirical claims of the paper and hope you feel similarly.

Reviewer comment (Writing):
The introductory statement of the results section should be reframed, as the reviewer does not consider any of the domains studied in this work to match the description of ‘real-world’ domains.

Author response:
**While I appreciate the goals of the system are to work for 'real-world’ domains, I would consider reframing the introductory statement of the results section, as I wouldn’t consider any of the domains studied in this work to match that description.**

Fair point! We will update the submission accordingly to remove that language.

Reviewer comment (Presentation):
More effort should be made to clarify that STITCH is not a contribution of this work. While this is clear to a careful reader, it should be made obvious. Additional citations to STITCH in Figure 1 and the respective paragraph in Section 3 would be appropriate.

Author response:
**I think a bit more effort should be made into clarifying that STITCH is not a contribution of this work – this is clear to a careful reader, but should be made obvious. Additional citations to STITCH in Figure 1 and the respective paragraph in Section 3 would seem appropriate.**

Thank you for the suggestion—we will add these additional citations to Stitch in the locations you suggested.

Reviewer comment (Presentation):
Table 1 appears to have a minor bolding issue in the CLEVR mean column (bottom half). More generally, the choice to bold only the highest number in the table should be reconsidered, as the standard deviations overlap greatly in many cases.

Author response:
**It looks like Table 1 has a minor bolding issue in the CLEVR mean column, bottom half. More generally, I’m unsure if bolding only the highest number is the right thing to do in this table, when the standard deviations have a great deal of overlap in many cases.**

Thank you for spotting this bolding issue—this has been addressed. Regarding overlap in results, this is something that other reviewers have also noted. We’ve gone ahead and underlined all results within 1 standard deviation of the best (bolded) result in each category.

Reviewer comment (Presentation):
While the performance differences between LAPS and LILO are discussed in A.5, the reviewer questions why this comparison is not presented directly in either Table 3 or Table 4.

Author response:
**While the performance differences between LAPS and LILO are discussed in A.5, is there a reason that this comparison isn’t put directly into either Table 3 or Table 4?**

(Assuming you’re referring to Table 1) This was primarily due to space and other practical considerations—the prior work from LAPS (Table 3) contains many different rows/ablations that one might reasonably want to compare between and cannot all fit inside Table 1. We also note that only a subset of the metrics that we report in Table 1 were reported in prior work. For instance, in Table 3, the *max* column is missing from REGEX and CLEVR, and standard deviations were not reported, so integrating that table into Table 1 would leave some gaps. We reasoned the simplest solution presentationally would be to keep all of the results from this work in one table and preserve the LAPS table in its entirety in the Appendix. If there’s extra space in the submission or if you have strong feelings on the issue, we can consider moving it over.