Paper: $\mathcal{B}$-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis (ICLR-2024)

Reviewer comment (Evaluation):
The authors filter programs at evaluation time using a different reward than the one used during training. This makes it hard to interpret their performance numbers. Could it be the filtering that is mostly responsible for them?

Author response:
> The authors filter programs at evaluation time using a different reward than the one used during training. This makes it hard to interpret their performance numbers. Could it be the filtering that is mostly responsible for them?

We hope our response to the main question helps address the concern about the use of a different reward. Ranking indeed improves performance as we expected. However, both initialization and conservative updates contribute significantly to the final performance. The comparison with CodeRL and RLTF with post-processing, i.e. critic sampling, is also available in Table 2. And we would like to highlight that our reward ranking is a more realistic post-processing strategy as explained above, and our model was trained with minimal reward designs.

Reviewer comment (Presentation):
The paper has a bit of an idiosyncratic structure, with e.g. a whole section dedicated only to the reward filtering at evaluation time, a very small Related Work section, and no outline of the paper's contents in the introduction, which makes it a bit hard to read.

Author response:
> - The paper has a bit of an idiosyncratic structure ...
> - Some more proof-reading is needed as typos can be found here and there.

We appreciate for the reviewer's constructive comments on our paper's presentation. We are delighted to implement these suggestions in the next revision, including but not limited to (1) adding an outline paragraph following the introduction for enhanced structure and clarity; (2) restructuring the related work section to ensure balanced and proportional content; and (3) conducting thorough proofreading to eliminate typographical and grammatical errors.

Reviewer comment (Writing):
Some more proof-reading is needed as typos can be found here and there.

Author response:
> - The paper has a bit of an idiosyncratic structure ...
> - Some more proof-reading is needed as typos can be found here and there.

We appreciate for the reviewer's constructive comments on our paper's presentation. We are delighted to implement these suggestions in the next revision, including but not limited to (1) adding an outline paragraph following the introduction for enhanced structure and clarity; (2) restructuring the related work section to ensure balanced and proportional content; and (3) conducting thorough proofreading to eliminate typographical and grammatical errors.

Reviewer comment (Evaluation):
Main question and concern: why do the authors use a filtering procedure at evaluation time based on a different reward model $\tilde{r}(s,a)$ than the environmental reward $r(s,a)$ they use for training? Given that the real reward $r$ can be easily and cheaply computed (a point which the authors themselves make in the introduction), I cannot see the need for such a procedure, and the authors do not elaborate on it. The authors should either: justify this choice; recompute their performance numbers with filtering based on $r$ at evaluation time; or re-train their method with the modelled reward $\tilde{r}$ from the outset.

Author response:
> Main question and concern: why do the authors use a filtering procedure at evaluation time based on a different reward model $\tilde{r}(s, a)$ than the environmental reward $r(s, a)$ they use for training? Given that the real reward $r$ can be easily and cheaply computed

We would like to response to the main question first, to clarify our choice of a different reward function $\tilde{r}(s, a)$ instead of the ground truth reward. 

The primary reason is the **availability of test cases**, which provide true rewards. During model training or evaluation, test cases are often available and required, as both training and test datasets are typically constructed with them. This context is what we meant by `real reward $r$ can be easily and cheaply computed'. However, **in practical scenarios, test cases are not always available**, making it challenging to obtain real rewards.

**An example scenario.** When **deploying a fine-tuned model for end-user applications**, users often provide natural language descriptions of their needs but may not include test cases (which can also be challenging for beginners or casual users). This situation is quite common and **real rewards are unobtainable**.

**$\tilde{r}$ in such scenarios.** Our reward model $\tilde{r}$ doesn't require test cases from end users, and hence is applicable. Moreover, in many applications, such as GitHub Copilot, the model must offer a single best response, not a range of options. Thus, the ability to **rank programs without test cases (true rewards)** is not only realistic but also advantageous.

**Additional benefits.** As discussed in Section 4, $\tilde{r}$ requires no additional training and adds minimal computational overhead, as $Q$ and $V$ are already evaluated during program generation. In contrast, the critic sampling strategy used in CodeRL and RLTF requires (i) training a separate repair model and (2) available test cases (i.e. true rewards) for post-processing, which is less desired due to aforementioned reasons.

**Remark on training with $\tilde{r}$.** Recall that our reward model $\tilde{r}$ is obtained from $Q$ using Eq (19), without additional training. Using $\tilde{r}$ to train $Q$ from outset is in fact creating a chicken-and-egg problem. Besides, complex reward engineering during training is also something we tried to avoid. 


We hope our explanation addresses the reviewer's major concern and clarifies our choice.

Reviewer comment (Experiments):
Could the mixed performance numbers be due to the conservative Bellman operator, which keeps the policy close to the initial checkpoint? Did the authors consider alternative definitions for $q(a|s)$, or to update it at some point during training?

Author response:
1. We believe the reward engineering is a major factor, though the trade-off made by our conservative Bellman operator might also be one of the reasons. Polyak update of $q$ with small/conservative update rate can be a good practice we believe. The key insights we seek to convey are: (1) the reasons why Bellman optimality operator is not particularly practical, and (2) the rationale behind how conservative backups address this issue, which we believe are novel and non-trivial to the program synthesis task. However, it's always possible to further refine the trade-off between conservatism and performance, along with other engineering aspects.

Reviewer comment (Theory):
Value-based methods, including DQN, can suffer from the so-called "Deadly Triad" of deep RL. Can the authors elaborate on how they avoid this pitfall?

Author response:
2. (**TL;DR**. The conservative operator avoids bootstrapping with overly optimistic value estimation. In addition, bounded value function and low initial TD-error (due to initialization) might also be helpful.)

    For the possible divergence caused by *Deadly Triad*, many techniques used in current deep RL era aims to reduce the **overestimation bias**[1]. For example, Double DQN *``reduce overestimations by decomposing the max operation in the target into action selection and action evaluation''* [2]; TD3 [3] proposes clipped double Q-Learning along with some other modifications. When training with a pre-colleted dataset, especially with partial coverage, recent offline RL works **avoid bootstrapping with OOD actions**, (OOD denotes out-of-distribution), as those unseen actions could have arbitrarily large estimations, leading to diverging behavior. For example, TD3+BC [4] simply adds an regularization term penalizing deviation from the dataset.

    Our methods resembles these above ideas: (1) we bound the value function by $R_{max}$, see Appendix D; (2) our conservative operator only select those actions with high probabilities $a'$ to evaluate target $Q(s', a')$, avoiding bootstrapping with OOD actions; (3) Our initialization may also alleviate this problem because it ensures value function starts with low bootstrapping error.

Reviewer comment (Reproducibility):
It seems that the authors add a value head $V^r_\theta(s)$ (besides its LM head) to their LLM. How is that done, and can it be done with a generic LLM?

Author response:
3. Yes, it is indeed possible for a generic LLM. Adding an additional head can simply done by adding a linear layer subsequent to the decoder, akin to how the LM head is added. As the reviewer was asking $V^r(s)$, we assume the reviewer meant to ask how to add a residual head initialized to output zeros. To achieves this, one can add two heads $h_1$ and $h_2$ and let $h_1$ be trainable, $h_2$ be fixed, setting $V^r = h_1 - h_2$ achieves this functionality. We hope that we have correctly understood the question and  our answer clarifies it.

Reviewer comment (Presentation):
In figure 2, it seems that the pass@1 metric is actually at its best at very beginning of training and that it never recovers afterwards. Is the figure only meant to be illustrative, or are those the real training curves?

Author response:
4. Those are the actual training curves. We hypothesize that it is because the initial checkpoint has been already fine-tuned on the training set, hence having a good training performance. Nonetheless, it highlights the difficulty of value-based learning and necessity of our proposed components.

Reviewer comment (Presentation):
Can the authors outline their paper's content and structure at the end of the intro?

Author response:
5. Certainly, we are delighted to add an outline of paper organization at the end of our introduction section in revision, to improve our paper's clarity.

Reviewer comment (Presentation):
In figure 3, it would be helpful to explicitly define what $\ell$ and $p$ stand for.

Author response:
6. We are grateful for the suggestion and will accordingly update the caption to explicitly define $\ell$ and $p$.

Reviewer comment (Presentation):
Why is the Nucleus Sampling outlined at the end of section 3? In my opinion it would make more sense to have in section 5, since it appears to be part of the evaluation protocol. Or is it also used during training?

Author response:
7. Nucleus sampling was not used during training. We agree with the reviewer that it would be more suitable in Section 5 and will move it accordingly.

Reviewer comment (Presentation):
It would be helpful if the authors defined (e.g. in a caption) what "Intro", "Inter", "Comp", and "All" mean in tables 1 and 2. While the text does mention it, it would be helpful to readers to be more explicit.

Author response:
8. Certainly, we will explicitly define the abbreviations in the table captions.

Reviewer comment (Experiments):
Why do the authors not consider $k=1000$ in table 2, as they do in table 1?

Author response:
9. The main reason that pass@1000 is missing in Table 2 is that they were not reported by the baselines, hence we cannot directly use the results. Evaluate pass@1000 for baselines are in fact very resources-intensive, especially considering that APPS has 5000 testing problems. As our resources were limited, we opted not to include the pass@1000 metric.

Reviewer comment (Presentation):
The study on generalisation of the reward model (notwithstanding my concerns on its use, detailed above) would better be moved to an appendix.

Author response:
10. We hope our response to the primary concern addresses the reviewer's doubts regarding our use of the reward model $\tilde{r}$. We believe that our reward model provides practical benefits, given the aforementioned explanation. Nevertheless, we are happy to move part of the content to the appendix, saving some space to accommodate the reviewer's suggestions on presentation.

Reviewer comment (Novelty):
Lack of evidence on why we need to do this to begin with.

Author response:
We would like to address the clarity and significance questions in a somewhat mixed order, as beginning with the technical definitions would help us better explain our motivations. We outline our responses as follows, 

1. Our response to (a) clarifies the technical distinctions between (on-policy) policy-based and (off-policy) value-based methods, highlighting the data inefficiency of the former due to their inability to reuse historical data, a limitation rooted in their technical nature.

2. Our response to (b) lists empirical evidence from RL literature and public RL benchmarks.

3. Our response to (c) provides a demonstrative example to validate the intuition discussed in (a).

4. The rest are responses to the reviewer's other concerns.

---

> (a) I feel the paper is overboard with definition and philosophy of "what is policy-based" and "what is value-based / RL". I believe these things have a very technical definition that the community can all agree with.

As pointed out by the reviewer, we do realize that our description was quite verbal, which could create ambiguity and confusion. Our intended focus is on the algorithm's capability to **re-use previously generated data**, as being unable to reuse old/historical data (costly to collect) is very inefficient. Mathematically, this can be characterized by determining whether the objective function requires **data to be drawn from the distribution induced by the current policy**, as demonstrated in the following examples:

| Algorithm | Category     | On/Off-Policy | Objective                                                                                                          |
|-----------|--------------|---------------|-----------------------------------------------------------------------------------------------------------|
| vanila PG | policy-based | on-policy     | $ L(\pi) = -\mathbb{E}_{\color{blue} \tau\sim\pi} [R(\tau)] $                                                      |
| PPO       | policy-based | on-policy     | $ L(\pi) = -\mathbb{E}_{\color{blue} \tau \sim {\pi_k}} [ \frac{\pi(s, a)}{\pi_k(s, a)}\hat{A}(s, a) ] $    |
| DQN       | value-based  | off-policy    | $ L(Q) = \mathbb{E}_{\color{blue} (s, a, s', r)\sim D} [(Q-\mathcal{B}^*Q)(s, a)]^2 $                              |
| Ours      | value-based  | off-policy    |  $L(Q) = \mathbb{E}_{\color{blue} (s, a, s', r)\sim D} [(Q-\mathcal{B}^q Q)(s, a)]^2 $                             |

where $R(\cdot)$ is cumulative trajectory-wise returns, $k$ denotes the current gradient/update step, $\tau \sim \pi$ or $\tau \sim \pi_k$ indicates the trajectories $\tau$ are rollouts collected by the **current policy** $\pi$ or $\pi_k$, whereas $D$ could be an **arbitrary dataset**.

This technical difference was not precisely reflected in our current writing, as commented by the reviewer. We are gratefully for the reviewer's suggestion on technical definitions, which will largely help us to improve our paper's clarity.

**Remarks on practical aspect.** (on-policy) policy gradient methods, in principle, cannot reuse previously generated, as they won't follow the current policy's (state-action) distribution. This is undesirable as generating samples with Transformers are known to be expensive.

PPOCoder indeed has some discussion on this aspect. To briefly recap, they generate synthetic programs at $k=0$ (prior to fine-tuning), and subsequently fine-tune their model using both human and synthetic programs drawn from this mixed distribution. It was observed in PPOCoder (Figure 3(e)) that *an increase in synthetic programs negatively impacts fine-tuning performance*. This is partly because the mixed distribution doesn't align with the fine-tuned policy's state-action distribution. 

However, with more data, an ideal objective is generally expected to either improve or at least maintain a policy's performance, not decrease it. In general, the use of (growing) historical off-policy data is the key to the better sample efficiency observed in off-policy value-based methods.

Reviewer comment (Experiments):
The statement from the conclusion section — "it is recognized being sample-inefficient (see e.g. Nachum et al., 2017), meaning poorly using off-policy data, even data previously generated" — is a very good motivation, but does this work actually demonstrate this with an experiment? It should, to show (in a table) that the proposed method can "learn more with less", with the x-axis showing the amount of training data, and the y-axis some performance metric. If this is indeed one of the experiments, it should be presented as the top and foremost contribution. Currently, the paper reads akin to "we did it because we can" but lacks the justification — in the form of an experiment — that this is something valuable to begin with.

Author response:
> (b) The following statement from the conclusion section is a very good motivation " it is recognized being sample-inefficient (see e.g. Nachum et al., 2017), meaning poorly using off-policy data, even data previously generated". Does this work actually demonstrate this with an experiment?

Although Nachum et al. [1] does not empirically demonstrate the sample inefficiency of policy-based methods, some empirical evidence could be found from other sources. For instance, [2] states that *``TD-style methods, such as off-policy actor-critic and Q-learning, are more sample-efficient ...''*, and supports this claim with empirical evaluation. 

Additional empirical comparisons can be found in public benchmarks, such as [OpenAI spinning up benchmark](https://spinningup.openai.com/en/latest/spinningup/bench.html), where policy-based methods often yield lower returns given the same number of environmental interactions. It can also be confirmed in some TD-learning works (e.g., [3, 4]), where policy-based baselines like PPO and TRPO often demonstrate lower sample efficiency.


> (c) I believe it should, to show (in a table) that the proposed method can "learn more with less", with x axis showing the amount of training data, and y-axis some performance metric.

We believe the suggested experiment on comparison of number of data will be very insightful. This unfortunately creates two practical difficulties: it requires re-training our model and the baseline models for different synthetic data budgets, which would be overwhelming to us due to time and computational constraints during the discussion period. More importantly, a fair comparison is only achievable using an identical reward function design, as the reward is certainly a major factor of training efficiency, e.g. [5, 6].



Nonetheless, we do believe that illustrating our motivation is important. To this end, we create a demonstrative experiment to illustrate empirical impact of re-using data with policy gradient objective. We hope the combination of 

- Response (a): an intuition given the technical definition 
- Response (b): empirical evidence in RL community, (showing value-based ones *``learn more with less''*)
- Response (c): an empirical validation of (a)
    
could addresses the reviewer's concern on our motivation.



**TL;DR**. The experiment below shows that using  policy gradient objective with off-policy dataset may even decrease the performance of a policy. It implies that one should use on-policy data (and discard historical data) in principle, contributing to the undesired sample inefficiency. Hence, the experiment empirically validates our intuition-level explanation provided in response (a).

Reviewer comment (Experiments):
If the authors are able to justify their work with some experimental evidence showing the advantages of value-based modeling, this work will be much improved.

Author response:
> (b) The following statement from the conclusion section is a very good motivation " it is recognized being sample-inefficient (see e.g. Nachum et al., 2017), meaning poorly using off-policy data, even data previously generated". Does this work actually demonstrate this with an experiment?

Although Nachum et al. [1] does not empirically demonstrate the sample inefficiency of policy-based methods, some empirical evidence could be found from other sources. For instance, [2] states that *``TD-style methods, such as off-policy actor-critic and Q-learning, are more sample-efficient ...''*, and supports this claim with empirical evaluation. 

Additional empirical comparisons can be found in public benchmarks, such as [OpenAI spinning up benchmark](https://spinningup.openai.com/en/latest/spinningup/bench.html), where policy-based methods often yield lower returns given the same number of environmental interactions. It can also be confirmed in some TD-learning works (e.g., [3, 4]), where policy-based baselines like PPO and TRPO often demonstrate lower sample efficiency.


> (c) I believe it should, to show (in a table) that the proposed method can "learn more with less", with x axis showing the amount of training data, and y-axis some performance metric.

We believe the suggested experiment on comparison of number of data will be very insightful. This unfortunately creates two practical difficulties: it requires re-training our model and the baseline models for different synthetic data budgets, which would be overwhelming to us due to time and computational constraints during the discussion period. More importantly, a fair comparison is only achievable using an identical reward function design, as the reward is certainly a major factor of training efficiency, e.g. [5, 6].



Nonetheless, we do believe that illustrating our motivation is important. To this end, we create a demonstrative experiment to illustrate empirical impact of re-using data with policy gradient objective. We hope the combination of 

- Response (a): an intuition given the technical definition 
- Response (b): empirical evidence in RL community, (showing value-based ones *``learn more with less''*)
- Response (c): an empirical validation of (a)
    
could addresses the reviewer's concern on our motivation.



**TL;DR**. The experiment below shows that using  policy gradient objective with off-policy dataset may even decrease the performance of a policy. It implies that one should use on-policy data (and discard historical data) in principle, contributing to the undesired sample inefficiency. Hence, the experiment empirically validates our intuition-level explanation provided in response (a).

Reviewer comment (Writing):
The paper goes overboard with definitions and philosophy of "what is policy-based" and "what is value-based / RL". These terms have technical definitions that the community can agree on. A cleaner story could have been: "Off-policy learning through value iteration for generation would be good, as it uses less data. Here's how to do it, and here's an experiment showing how well we can do as a function of the amount of human data required."

Author response:
> (a) I feel the paper is overboard with definition and philosophy of "what is policy-based" and "what is value-based / RL". I believe these things have a very technical definition that the community can all agree with.

As pointed out by the reviewer, we do realize that our description was quite verbal, which could create ambiguity and confusion. Our intended focus is on the algorithm's capability to **re-use previously generated data**, as being unable to reuse old/historical data (costly to collect) is very inefficient. Mathematically, this can be characterized by determining whether the objective function requires **data to be drawn from the distribution induced by the current policy**, as demonstrated in the following examples:

| Algorithm | Category     | On/Off-Policy | Objective                                                                                                          |
|-----------|--------------|---------------|-----------------------------------------------------------------------------------------------------------|
| vanila PG | policy-based | on-policy     | $ L(\pi) = -\mathbb{E}_{\color{blue} \tau\sim\pi} [R(\tau)] $                                                      |
| PPO       | policy-based | on-policy     | $ L(\pi) = -\mathbb{E}_{\color{blue} \tau \sim {\pi_k}} [ \frac{\pi(s, a)}{\pi_k(s, a)}\hat{A}(s, a) ] $    |
| DQN       | value-based  | off-policy    | $ L(Q) = \mathbb{E}_{\color{blue} (s, a, s', r)\sim D} [(Q-\mathcal{B}^*Q)(s, a)]^2 $                              |
| Ours      | value-based  | off-policy    |  $L(Q) = \mathbb{E}_{\color{blue} (s, a, s', r)\sim D} [(Q-\mathcal{B}^q Q)(s, a)]^2 $                             |

where $R(\cdot)$ is cumulative trajectory-wise returns, $k$ denotes the current gradient/update step, $\tau \sim \pi$ or $\tau \sim \pi_k$ indicates the trajectories $\tau$ are rollouts collected by the **current policy** $\pi$ or $\pi_k$, whereas $D$ could be an **arbitrary dataset**.

This technical difference was not precisely reflected in our current writing, as commented by the reviewer. We are gratefully for the reviewer's suggestion on technical definitions, which will largely help us to improve our paper's clarity.

**Remarks on practical aspect.** (on-policy) policy gradient methods, in principle, cannot reuse previously generated, as they won't follow the current policy's (state-action) distribution. This is undesirable as generating samples with Transformers are known to be expensive.

PPOCoder indeed has some discussion on this aspect. To briefly recap, they generate synthetic programs at $k=0$ (prior to fine-tuning), and subsequently fine-tune their model using both human and synthetic programs drawn from this mixed distribution. It was observed in PPOCoder (Figure 3(e)) that *an increase in synthetic programs negatively impacts fine-tuning performance*. This is partly because the mixed distribution doesn't align with the fine-tuned policy's state-action distribution. 

However, with more data, an ideal objective is generally expected to either improve or at least maintain a policy's performance, not decrease it. In general, the use of (growing) historical off-policy data is the key to the better sample efficiency observed in off-policy value-based methods.

Reviewer comment (Writing):
Defining program synthesis as "generating program from natural language" is imprecise. Other works define "program synthesis" differently, such as generating a program from input-output examples alone, or using constraint solvers to generate programs from correctness specifications. A better wording would be beneficial, as the current narrow opening definition may cause a negative reaction.

Author response:
> Defining program synthesis as "generating program from natural language" is imprecise

Thank you for pointing it out. We realize that we were narrowly focused in terms of program synthesis definition, because we were somewhat limited to the specific APPS dataset. It is very true that the program synthesis problem should be more general. We are delighted to revise our definition to be more accurate and encompassing, in line with the reviewer's suggestion.

Reviewer comment (Evaluation):
There should be a report on the average time it takes to sample a full program, compared against some other policy-based methods.

Author response:
> Time to take sample?

We take 5 problems from APPS test set (id 0 - 4) as examples and sampled 1000 programs for each, using one RTX 4090 24GB graphics card, with maximum batch sizes that do not create OOM errors. The table below shows the sampling time (in secs) per 1000 programs, in the format of mean$\pm$std. Our method takes approximately 1.7 to 2.0 times longer to sample. 

| Temperature | CodeRL (batch size = 66) | Ours (batch size = 44)|
|---------|---------|---------|
| 0.4 | 291.18$\pm$56.08 | 495.78$\pm$189.35 |
| 1.0 | 290.88$\pm$56.20 | 595.84$\pm$65.66 |


**Remark on faster sampling.** While we originally proposed using $Q$ as logits for sampling, it is also feasible to use the advantage $A$ as logits for sampling, especially when the goal is to generate as many outputs as possible without evaluating $Q$ and $V$ values. This would not require evaluating $V$, and therefore, we would expect similar sampling times. Under the same setting, sampling with advantage $A$ takes $275.79$ seconds and $322.78$ seconds for temperatures of $0.4$ and $1.0$, respectively, which are similar to CodeRL as we expected.