Paper: Generating Pragmatic Examples to Train Neural Program Synthesizers (ICLR-2024)

Reviewer comment (Experiments):
This paper only compares the pragmatic framework against the literal and HFT baselines, which are derivatives of the pragmatic framework. While there is a comparison against GPT3.5, I would have liked to see other program synthesis baselines that do similar things for a more thorough comparison.

Author response:
> This paper only compares the pragmatic framework against the literal and HFT baselines, which are derivatives of the pragmatic framework. While there is a comparison against GPT3.5, I would have liked to see other program synthesis baselines that do similar things for a more thorough comparison.

In section 1 of the General Response, we detail why the Literal model is not derivative of our pragmatic approach, and in itself a standard program synthesis baseline. In section 3 we describe some findings from running a symbolic regex-specific program synthesizer.

Reviewer comment (Reproducibility):
Will the dataset be released?

Author response:
> Will the dataset be released?

Yes! It’s included in the supplement.

Reviewer comment (Reproducibility):
What distribution do you use to sample the set of rows and columns to update for RSA?

Author response:
> What distribution do you use to sample the set of rows and columns to update for RSA?

We use the listener model to generate programs consistent with the prior example (“ab”, ✓) and the speaker model to generate examples consistent with the target program +b* as shown in Figure 1 of the paper. So, for the given example, in the second turn of the simulated interaction we may get a set of programs [a?b?, a+b+, a*b+c?] and examples [(“aab”, ✓), (“abb”, ✓), (“aa”, ✓), (“b”, ✗)], and use those to construct the consistency matrix over which we perform RSA inference.

Reviewer comment (Theory):
It appears that one assumption of RSA is that given a certain example, the literal listener $L_0$ should assign equal probability to all programs consistent with it. This is unlikely to be the case once $L_0$ is a neural net, unless this has been perfectly trained.

Author response:
This is correct – it is not possible to model a uniform distribution over all consistent programs using the literal listener. We also added a prior term to the description of the L0 computation, thanks for pointing it out. This prior is implicitly modeled by the base model when it is trained. However, this non-uniform prior is used only when we sample from the base model. We follow prior work (Pu et al., 2020) and explicitly impose a uniform prior over the sampled programs (that we use to populate the consistency matrix we perform RSA inference over). We have amended the text of Section 3.2 to convey this more clearly.

Reviewer comment (Theory):
As mentioned above, the RSA framework assumes $L_0$ to assign equal probability to all consistent programs, which is not going to be the case once it is approximated with a Neural Net. Could the authors comment on this?

Author response:
This is correct – it is not possible to model a uniform distribution over all consistent programs using the literal listener. We also added a prior term to the description of the L0 computation, thanks for pointing it out. This prior is implicitly modeled by the base model when it is trained. However, this non-uniform prior is used only when we sample from the base model. We follow prior work (Pu et al., 2020) and explicitly impose a uniform prior over the sampled programs (that we use to populate the consistency matrix we perform RSA inference over). We have amended the text of Section 3.2 to convey this more clearly.

Reviewer comment (Writing):
The literal listener $S_0$ is not defined in section 2. Only $L_0$ and $S_1$ are defined. Could the authors provide a definition?

Author response:
We amended the paper to include the definition of $S_0$ analogous to $L_0$: $S_0(example|program) \propto M(example, program)P(example)$

Reviewer comment (Experiments):
Why didn't the authors include any baselines (neural or not) specific to the task of inferring regexs?

Author response:
In section 3 of the General Response we describe some findings from running a symbolic regex-specific program synthesizer.

Reviewer comment (Evaluation):
Some details of the evaluation protocol are a bit obscure. First of all, how do the authors compute their top-1 metric on the validation dataset, when doing model selection? They only provide a definition of the metric as part of the final human trial, so it's not clear how the model would be prompted when computing it at the model selection stage.

Author response:
Here is how we perform model selection. Given a model, we present the examples (a single string-label pair) from the validation set one example at a time, in sequence, and count the fraction of instances in which the model recovers the correct program. This fraction is used to score the models. This is similar to the replay setting which we use to evaluate GPT-3.5.

Reviewer comment (Writing):
The authors state that "TOP-1@$t$ measures whether the model's top-1 matches intended regular expression at any point at turn $t$ of the interaction". Is it at any point, or a turn $t$? Since the interaction ends after a match is achieved, it can only be at turn $t$, but the text is ambiguous.

Author response:
Yes, this is a typo, thanks for pointing it out! We mean that for a turn t, the TOP-1@t metric measures the fraction of tasks solved after t turns are complete (that is, the user has provided t examples).

Reviewer comment (Reproducibility):
The authors detail their inference procedure in section 4.4. Therein they state that programs inconsistent with a given example are filtered out. Shouldn't they be left in in order to build the consistency matrix $M$? Does this paragraph only detail the inference protocol for the evaluation phase, or also for the training phase? This is not clear.

Author response:
The protocol is for both training and inference. We want the model to learn how to disambiguate between consistent programs, assuming that the base model is already effective at finding a consistent program. Note that the programs sampled are consistent with the given specification, but may be inconsistent with other examples sampled by the speaker that are not yet added to the specification, which builds a consistency matrix with both 1s and 0s.

Reviewer comment (Writing):
I assume that the numbers reported in the tables refer to the _fraction_ of instances in which an interaction was successful at time $t$, so it's technically incorrect that the metrics measure "whether" something happens. They actually measure "how often" it happens over a set of interactions. It would be helpful to the reader to amend this somewhat inaccurate language.

Author response:
We will amend the explanation of the method to make this clear.

Reviewer comment (Writing):
From the intro: "A synthesizer trainer in the style of Devlin et al...." what does this mean? Could the authors be more explicit?

Author response:
We elaborate on this in section 1 of the General Response. We have also updated section 3.2 to make this more clear.

Reviewer comment (Reproducibility):
It is unclear exactly how conditioning on previous examples is done when sampling in terms of the consistency matrix; further clarification is needed.

Author response:
## Conditioning on previous examples
> I also don't understand exactly how the conditioning on previous examples are done when sampling in terms of the consistency matrix. Could you elaborate on that?

When generating examples at each turn, we resample the rows (programs) and columns (examples) based on the examples already added to the specification. This allows us to look at a relevant sample of the consistency matrix, as explained above. 
Referring to Figure 1 of the paper, we unroll an interaction between the speaker and listener models over a sequence of turns. In the first turn, there are no prior examples. The speaker generates examples only based on the target program (in this case a+b*), and the listener synthesizes programs that satisfy an empty specification. We then perform RSA inference and select an example to add to the specification. In the figure, this example is shown to be (“ab”, ✓). For the next turn, we use the specification built so far (now [(“ab”, ✓)]) as inputs to the speaker and listener models, and sample the next example as shown in the figure. This example is added to the specification, and the process is repeated.

Reviewer comment (Novelty):
There is a connection to be made between the proposed method and bootstrapped “wake-sleep” approaches for program synthesis [1,2]. Both learn “generative” and “inference” models that learn on one another’s outputs. The modeling set-ups are different, as these wake-sleep approaches move towards a target distribution, whereas the proposed method optimizes for synthetic training data that matches a prior desiderata (pragmatic I/O examples), but these ideas are close enough that they should be discussed within the related work section.

Author response:
We have also added some discussion about other iterated bootstrap training to the paper to highlight these connections to the literature.

Reviewer comment (Experiments):
The proposed system effectively improves the listener model by finding “better” synthetic training I/O examples, where better means there is a pragmatic connection between the examples and the target program. However, it’s unclear if this improvement changes the upper-bound of the listener model performance, or if it just helps the listener model reach a good performance with less training iterations. Training the base model for 300k programs, for a single epoch, it's not clear whether the model has started to plateau in performance. It would be helpful to provide evidence that the base model has saturated, by e.g. plotting validation performance over pretraining iterations. What would this plot look like?

Author response:
Figure 6 in the updated draft plots the validation metric (Top-1 success) on the validation set (40 program-specification pairs). We see that the base model has started to plateau in performance after 1 epoch of training (9375 steps). We have also updated Appendix B with this experiment.

Reviewer comment (Experiments):
It’s also not clear to me why starting from a pretrained model like ByT5 would be necessary or helpful. The programs come from a constrained DSL, where unlimited synthetic data can be sampled, so it should be possible to train the base models from scratch. It would be good to include an ablation on how starting with or without pretraining affects the proposed method. What is the justification for starting with a pretrained model?

Author response:
We also observe a substantial gain in sample-efficiency from using a pretrained model (as seen in the plot in Figure 6 of the updated draft). But, in general we agree with your statement that with the ability to generate unlimited synthetic data, we expect that we can start with a randomly initialized model and train until we are able to match the performance of starting with a pretrained model. However, since our goal here is to have a strong base synthesizer (while being agnostic to the specific type of base synthesizer), we opted to use a pretrained model in the interest of sample-efficiency.

Reviewer comment (Evaluation):
Much of the evaluation is based off of human-interactions, which is present only in limited quantities. Are there any metrics which could be evaluated without human interactions? For instance, what about the following set-up:  
1. Pick a target regular expression and a single I/O example at random  
2. The listener samples a target expression given the current specification  
3. With respect to (2), the speaker samples an example, which is annotated as consistent/inconsistent by an oracle  
4. Repeat steps 2 and 3 until the listener samples the *correct* target expression  
The metric would then be the number of example generations needed for the listener to predict the correct regular expression, where the idea would be that as the speaker is better at producing "pragmatic" examples it should require less steps versus a baseline that for instance randomly sampled an example. Beyond serving an evaluation set-up that requires no human-data, this kind of framework could conceivably even be useful for “real-world” applications: e.g. this could reduce the burden on an end-user, who instead of having to think up new examples as input, would just need to label them.

Author response:
Since the goal of building this kind of synthesizer is to better interpret human communication, the most faithful evaluation is to have humans interact with this kind of system. Having a model stand in for a human in evaluation might not be representative of the results of human interaction. With specific reference to an algorithm like the one you proposed, the choice of a random I/O example at the first turn is already a significant deviation from human behavior, since a human’s first example would also be chosen informatively. This would in turn influence every future turn of interaction too.