CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
Anonymous Author(s)
AffiliationAddress
Abstract
Program synthesis or code generation aims to generate a program that satisfies aproblem specification.
Recent approaches using large-scale pretrained languagemodels (LMs) have shown promising results, yet they have some critical limitations.
In particular, they often follow a standard supervised fine-tuning procedure to traina code generation model from natural language problem descriptions and groundtruth programs only.
Such paradigm largely ignores some important but potentiallyuseful signals in the problem specification such as unit tests, which thus resultsin poor performance when solving complex unseen coding tasks.
To address thelimitations, we propose “CodeRL”, a new framework for program synthesis tasksthrough pretrained LMs and deep reinforcement learning (RL).
Specifically, duringtraining, we treat the code-generating LM as an actor network, and introduce acritic network that is trained to predict the functional correctness of generatedprograms and provide dense feedback signals to the actor.
During inference, weintroduce a new generation procedure with a critical sampling strategy that allows amodel to automatically regenerate programs based on feedback from example unittests and critic scores.
For the model backbones, we extended the encoder-decoderarchitecture of CodeT5 with enhanced learning objectives, larger model sizes andbetter pretraining data.
Our method not only achieves new SOTA results on thechallenging APPS benchmark, but also shows strong zero-shot transfer capabilitywith new SOTA results on the simpler MBPP benchmark.
Introduction
Considering program synthesis as a sequence-to-sequence task pretrained language models (LMs)[Hendrycks et al.,
Chen et al.
2021a
Austin et al.
2021] can be adapted to receive inputsequence as problem specification in natural language and generate a sequence of codes as theoutput program (See Figure 1, right, for an example).
While these models achieve promising results,especially in basic programming tasks [Chen et al., 2021a, Austin et al., 2021], we observe that theystill fail to generate codes to solve complex problems [Hendrycks et al., 2021, Li et al., 2022].
There are two main limitations.
First, current models are trained using a conventional next-tokenprediction (NTP) objective which maximizes the next ground-truth token likelihood.
As noted in NLPdomains Bengio et al. [2015], Ranzato et al. [2016], training models only with next-token predictionobjective in a "teacher-forcing" manner often leads to accumulating errors during test time whentokens are generated by conditioning on previously sampled tokens, not the ground-truth tokens.
This issue becomes more serious in the domain of program synthesis, where token-matching scores such
Actor-Critic RL Finetuning with
Returns
Problem Specification
A string is a palindrome if it reads the same from the left to the right and from the right to the left….If
there is such a substring in $s$ that is not a palindrome, print the maximum length of such a substring….
Example Input and Output:
Input: ‘hannah’
Output:
Solution Program
Unit Tests Input: wuffuw
Output: 5 Input: iiiiiii
Output: 0…
[Figure]
as BLEU [Papineni et al., 2002, Ren et al., 2020] are more appropriate in partial program synthesistasks (i.e. code completion)
[Husain et al., 2019] but have failed to measure the functional correctnessof complete programs [Hendrycks et al., 2021, Chen et al., 2021a].
Training only with NTP objectiveis hence, not ideal to tackle full program generation to solve programming problems.
Secondly, current models fail to utilize the potential meaningful signals from unit tests, which directlydetermine the model performance by the functional correctness of programs.
Current approachesneglect this important signal during model optimization as well as generation procedure.
Duringoptimization, unit tests could be factored into learning objectives to match the final goal of generatingsemantically correct programs.
During inference, since unit tests are often parts of problem description(i.e.
example unit tests), they are potentially powerful to further improve output programs.
To address the above issues, we introduce “CodeRL”, a new framework to improve pretrained LMs forprogram synthesis tasks through reinfocement learning (See Figure 1, left).
Specifically, we propose atraining strategy that optimizes pretrained LMs for program synthesis tasks in an actor-critic approach[Konda and Tsitsiklis, 1999, Sutton et al., 1999].
We treat the pretrained LM as an actor networkand synthetically sample sequences from this actor, including both correct and incorrect programs.
These program samples are passed to a critic model which is trained as an error predictor to assessthe functional correctness of these samples.
We use the token-level hidden states extracted from thelearned critic model to estimate the values/scores of output tokens of these synthetic samples.
Theactor network is then finetuned on these synthetic samples weighted by their critic scores.
Duringinference, we introduce a new generation procedure that involves example unit tests and a criticto filter and select sub-sequences.
These sub-sequences are utilized as seeds which condition themodel to resample new tokens and obtain new output programs.
This approach allows the model toautomatically refine output programs based on their functional correctness during test time.
We extend CodeT5 with better pretraining strategies as the foundation model for CodeRL.
Ourcomprehensive experiments show that our models can achieve SOTA performance on the challenging
APPS benchmark [Hendrycks et al., 2021].
Specifically, our models reach more than 2% pass@1 ,6% pass@5 , and 19% pass@1000 .
Since our RL method is model-agnostic, we apply it to variouslarge-scale models and achieve consistent performance gains.
We further test its zero-shot transferability on a simpler MBPP benchmark [Austin et al., 2021], where it sets a new SOTA result of 63.0%pass@80 over a finetuned GPT-137B’s 61.4%.
We release the improved CodeT5-large ( 770 M) modelwhich outperforms many pretrained LMs of much larger sizes.
Related Work
Program Synthesis.
Program synthesis tasks can date back as early as the early adoption ofmachine learning research Waldinger and Lee [1969], Manna and Waldinger [1971].
Earlier tasksinclude problem specifications in the form of input-output (IO) examples [Summers, 1977, Gulwaniet al., 2012] and synthesis methods are limited to probabilistic approaches [Liang et al., 2010] or simple programming concepts [Joulin and Mikolov, 2015, Kurach et al., 2015].
As deep learningmethods became popular, later approaches adopt neural models to induce output programs, assumingan inductive bias given large number of program samples [Parisotto et al., 2016, Balog et al., 2016,
Devlin et al., 2017].
More recently, we witnessed the emergence of program synthesis tasks in whichoutput programs are extended to general-purpose programming languages
[Yin and Neubig, 2017, Xuet al., 2018, Chen et al., 2021a] and program specifications are fully described in natural English text[Hendrycks et al., 2021, Austin et al., 2021, Poesia et al., 2022].
These extensions have encouraged arising number of applications of pretrained language models (LMs) to program synthesis to exploitthe contextual representations learned from massive data of codes and natural languages
[Feng et al.
,2020, Clement et al., 2020, Wang et al., 2021, Wang and Komatsuzaki, 2021, Chen et al., 2022].
Despite impressive results in basic programming problems and initial commercial deployment 1 ,existing models still perform poorly against complex problems such as those from programmingcompetitions on Codeforces [Hendrycks et al., 2021, Li et al., 2022].
Reinforcement Learning for Sequence Generation.
Related to the program synthesis tasks areresearch domains of sequence generation, in which RL approaches have demonstrated remarkableachievements.
In these domains, RL approaches are used to exploit signals from non-differentiablemetrics of the task at hand.
Earlier work such as [Ranzato et al., 2016] adopts this strategy with
REINFORCE algorithm
[Williams, 1992] to directly optimize models for sequence-based test metricssuch as BLEU [Papineni et al., 2002] and ROUGE
[Lin, 2004] scores for translation models.
In thesame domain, Bahdanau et al.
[2016] introduced an actor-critic framework [Sutton, 1984, Konda and
Tsitsiklis, 1999].
In visual captioning domains, Rennie et al.
[2017], Wang et al.
[2018] proposed touse RL to optimize image captioning models using variants of CIDEr scores [Vedantam et al., 2015].
Alternatively, Ren et al.
[2017] derived a new goal-oriented return estimate using visual-semanticembedding.
Johnson et al.
[2017], Trivedi et al.
[2021] introduce program generation as an auxiliarytask to learn interpretable policies in question-answering and synthetic navigation tasks.
Different from prior domains, in program synthesis, Austin et al.
[2021], Chen et al.
[2021a], Li et al.[2022] demonstrated very low correlation between token-based similarity metrics and functionalcorrectness of programs.
Hence, it is not trivial to define an appropriate optimization goal in thisdomain.
We propose to exploit unit test signals, which directly exhibit functional correctness ofprograms, during both - model optimization and test-time generation stages.
More related to our workare RL-based program synthesis [Guu et al., 2017, Bunel et al., 2018, Liang et al., 2018, Zhong et al.
and execution-guided synthesis approaches [Ellis et al., 2019, Chen et al., 2021b].
However,these are limited to programming languages defined within a specific application domain only.
CodeRL
3.1 Program Synthesis Task
Following a sequence-to-sequence approach, the program synthesis task contains a problem description as an input sequence D and an output sequence of program
ˆ W =
( ˆ w 1 , ..., ˆ w T ) , ˆ w t ∈ V 2 thatcan solve the problem.
The output at each decoding step t is a distribution over the vocabulary V ,computed by the softmax function ˆ w t ∼ softmax(Linear( s t )) where s t is the contextual hiddenstate at decoding step t .
Conventionally, during train time, model parameters, θ , are learned bymaximizing the likelihood of the ground-truth reference programs.
Denoting W =
( w 1 , ...w T ) asthe ground-truth program, the objective is to minimize the cross-entropy loss:
[Equation]
where the conditional probability p θ is parameterized following the above softmax function.
Duringtest time, models generate sequences of programs by autoregressively sampling token ˆ w t from thedistribution p θ ( .
| ˆ w 1: t − 1 , D ) .
Models are evaluated against unit tests corresponding to the problem.
[Figure]
Public
Each test includes a pair of input and ground-truth output.
In real-world program synthesis tasks[Hendrycks et al., 2021], example unit tests are often given as parts of the problem specification.
3.2 Pretraining Language Models on Code
We adopt Transformer models as the backbone of our program synthesis systems.
Specifically, thispaper extends the CodeT5 model [Wang et al., 2021] as a foundation model for CodeRL.
CodeT5.
CodeT5 [Wang et al., 2021] is a multi-lingual code-aware language model pretrained onlarge-scale source code corpora curated from Github.
With a unified encoder-decoder architecture,
CodeT5 achieves state-of-the-art performance in a wide range of code intelligence tasks in the
CodeXGLUE benchmark [Lu et al., 2021] including both code understanding and generation tasks.
Improving Pretraining Data.
We enlarge the Python pretraining dataset using the recently releasedlarge-scale Github Code dataset 3 .
We filter the dataset by keeping only the code with licenses thatat least permit academic use (“mit”, “apache-2”, “bsd-3-clause”, “bsd-2- 126 clause”, “cc0-1.0”,“unlicense”, “isc”).
The resulting Python dataset (GCPY) has 10.5B tokens and is 10x larger than the
CodeSearchNet (CSN) corpus [Husain et al., 2019] used in the original CodeT
[Wang et al., 2021].
Improving Pretraining Objective.
While pretraining tasks in CodeT5 like masked span prediction(MSP) benefit code understanding tasks, they have a large discrepancy with program synthesisobjectives.
To mitigate this gap, we introduce a pretraining task of next-token prediction (NTP) into
CodeT5.
Specifically, we uniformly sample a pivot location for each code sample, then pass thecontent preceding the pivot to the encoder and remaining to the decoder.
To control the length ofinput and output sequences, we restrict the pivot within 10% to 90% of the original sequence.
3.3 Program Synthesis as an RL Problem
We propose to formulate the Program Synthesis as an RL problem and apply an actor-critic RLapproach to improve the performance of a pretrained LM by exploiting the unit test signals in bothmodel optimization (See Fig. 2) and generation procedure (See Fig. 3).
More formally, we can view the learned parameters of an LM model, θ as a stochastic policy , whichdecides an action as the prediction of each token.
Following each action, an LM model updates itshidden state representations which are used by the policy to determine the next action in the nextdecoding step.
At the end of the generation episode (i.e. an <endoftext> token is observed), the LMmodel receives a return r measured by the functional correctness of the generated program.
The goalof RL finetuning is to minimize the expected return:
[Equation]
where W s = ( w s 1 , ..., w sT ) is a synthetic sample in which each token w st is sampled by the LM
modelat decoding time step t .
Following the REINFORCE algorithm
[Williams, 1992, Sutton and Barto,2018] and policy gradient theorem [Sutton et al., 1999] we can define an estimate of the gradient
∇ θ L ( θ ) of the non-differentiable return r as:
[Equation]
Defining Return by Unit Test Signals.
For each sample sequence W s , the return r can be defined heuristically by checking its functional correctness.
We pass generated programs together with the corresponding unit tests to a compiler.
From the outputs of the tests, we can determine the return r :
[Equation]
be compiled
[Equation]
.e.
compile error
[Equation]
cannot be executed
[Equation]
unit
[Equation] [Equation]
error
[Equation]
However, in related domains such as text-to-SQL research [Zhong et al., 2018, Xu et al., 2018], wenote that this approach to estimate returns can lead to unstable training process with high variance ofthe gradient estimate following Eq.
(3) with mini-batches in training.
Return with a Baseline.
In order to alleviate this variance, we adopt a “baseline” [Sutton and Barto,2018].
Specifically, we use a greedy decoding strategy as a baseline and any generated samplesthat outperform this baseline are given positive return estimation, and negative return estimationotherwise.
This relative normalization technique allows models to explore imperfect programs, aslong as their returns are better than the baseline’s.
Given a training sample, we denote the return ofthe baseline r ( W b ) and the expected gradient is computed as:
[Equation]
Note that at each decoding step t , our greedy decoding baseline is independent from the action w stand hence, the expected gradient term ∇ θ L rl ( θ ) from Eq.
(3) remains the same in Eq.
Intermediate Return by Critic as Error Predictor.
We observe that the above gradient estimateis only based on a final return at the end of the decoding process.
However, programs often followfixed syntactical rules in which a single token such as an additional white-space can render a programerroneous.
Therefore, Eq.
(8) becomes too restrictive.
A straightforward solution is to use tokenbased similarity scores [Papineni et al., 2002, Ren et al., 2020]) between each subsequence W s 1: t andthe ground truth.
However, code matching is not an ideal return measure due to its poor correlationwith program correctness [Hendrycks et al., 2021, Chen et al., 2021a, Austin et al., 2021] which canonly be measured against fully complete programs.
Alternatively, we introduce a critic model.
The critic model is parameterized as a neuralnetwork with parameters ϕ that receives inputs as the problem description D and a sampledprogram W s = { w s 1 , . . .
, w sT } .
The critic is trained to infer the unit test outcome; one of{ CompileError , RuntimeError , FailedTest , PassedTest } as described in the return definitions in
Eq.
(4) to (7).
The training objective of the critic ϕ can be expressed as:
[Equation]
where u denotes the ground-truth unit test outcome given by the compiler.
We use Transformermodels of smaller sizes than the actor model as the base architecture for the critic model.
Thecontextual hidden states of the program tokens ( h 1 , . .
. , h T ) obtained from the critic model decoderare max-pooled along the sequence length dimension h pool =
Pooling( h 1 , . .
. , h T ) .
The critic’sprediction on the unit test outcome is computed as ˆ u =
softmax(Linear( h pool )) .
Given a learned critic, we use the probability distribution ˆ v t =
softmax(Linear( h t )) to estimate thetoken-level value ˆ q of w st in relation to the ground-truth unit test output (note that we use the tokenlevel contextual representation h t here, before the pooling operation).
Specifically, ˆ q ϕ ( w st ) =
ˆ v t [ u ]where ˆ v [ . ] denotes the probability of a specific unit test outcome from the four possible ones.
We use
[Figure] [Figure]
[Figure]
this estimate to train the actor LM model with intermediate returns:
[Equation]
Generating Programs with Example Unit Tests and Critic.
We leverage the unit tests provided inthe input problem description to improve the generation procedure during inference too (see Figure 3for an overview).
For each problem, we generate N programs, out of which we only select programsthat pass example tests (leading to a set F ) and filter out the rest.
To improve sample quality, weperform another round of generation where we use sub-sequences from these filtered samples asprompts (or “seed” sequences) to the actor LM.
We employ a separate critic model ( ϕ test ) to guideour choice of sub-sequences from these filtered samples.
This critic model is trained with a similarobjective as Eq.
(9), but in a binary classification setup with { FailedTest , PassedTest } labels.
Let W filter =
{ w 1 , . .
. , w T } denote a generated sample that passes the example unit tests.
Weuse the critic model to assign a value to each token ˆ q ϕ test ( w t ) =
p ϕ test (ˆ u = PassedTest | w 1: t , D )corresponding to the critic’s predicted probability of the sub-sequence till t passing the unit tests.
We split the sequence at position t max corresponding to the highest critic assigned value and usethe left split as the seed for the next stage.
If this seed sequence till t max contains a token withp ϕ test (FailedTest) > p ϕ test (PassedTest) , we further chop it at this token by removing tokens on theright.
This is done to pick prompts that are likely to generate successful programs in the next round.
We use these seeds to initialize and condition the (actor) LM to resample new tokens till we encounterthe <endoftext> token.
In this round, each seed sequence can be stacked N/ |F| times for upsampling.
This results in the same number of output programs N .
We call this generation procedure as “Critic
Sampling” (CS).
We use mini-batch generating to improve efficiency during inference and employnucleus sampling with a batch size of N = 200 .
While we do incur additional cost to re-sampleusing the seed sequences, we are only required to generate partial programs in the re-generation stage,making this stage less expensive than conventional generating procedures.
Experiments
4.1 Experimental Setups and Datasets
Pretraining Setup.
We pretrain a CodeT5-large model (770M) from scratch following T5-large’sarchitecture [Raffel et al., 2020].
We follow the pretraining setups in CodeT5 [Wang et al.
, 2021]with the modifications as proposed in §3.2.
We evaluate this new pretrained CodeT5 model on
CodeXGLUE [Lu et al., 2021] and achieve new SOTA results (See the Supplementary).
APPS Benchmark.
We choose the challenging APPS program synthesis benchmark [Hendryckset al., 2021], as it has large coding problems of varying difficulties collected from multiple codingwebsites.
It includes training and test splits, each of which has 5000 samples of programming taskswith diverse levels of difficulty, including “Introductory”, “Interview”, and “Competition” levels.
Each sample includes 20 unit tests on average to validate the functional correctness of programs.
[Table] [Table]
Finetuning Setup.
Due to the potential large number of trajectories (i.e. V T ) to generate a sequenceand the unstable feedback loop between actor and critic [Lillicrap et al., 2015, Wang et al., 2018],we applied imitation learning to first warm-start a pretrained LM model with L ce only for up to 10epochs.
We then sampled sequences of program from this actor network to train the critic whilekeeping the parameters of the actor network frozen.
For experiments with CodeT5 actor models, weuse the CodeT5-small architecture for the critic model, and GPT2-small critic architecture when theactor models are GPT variants.
After training the critic, we then apply both L ce and L rl with equalweights to finetune the actor network.
Evaluation.
We follow [Hendrycks et al., 2021, Chen et al., 2021a] and evaluate the models usingthe pass@k metric, which is the percentage of problems solved by using k generated programs perproblem.
We also follow Li et al.
[2022] and use n@k metric which only considers a subset of ncandidates from k generated programs per problem.
The subset of n candidates are typically selectedby a filtering method by passing generated programs through example tests given as part of theproblem description [Chen et al., 2021a, Li et al., 2022].
For more details of experimental setup, please refer to the Supplementary Material.
4.2 Experimental Results on APPS
Baselines.
As reported by Hendrycks et al.
[2021], we compared our models with several baselines,including GPT2 [Radford et al., 2019],
GPT-Neo
[Black et al.], and GPT3 [Brown et al., 2020].
Wealso compare the results with Codex [Chen et al., 2021a] and AlphaCode
[Li et al., 2022].
Note thatby default, results of pretrained LMs (except for Codex and GPT3) are from models finetuned on
APPS using the standard loss L ce only.
In our ablations, since CodeRL is model-agnostic,
we canalso integrate it with GPT variants such as GPT-J [Wang and Komatsuzaki, 2021] and GPT-Neo.
Overall Results.
Firstly, Table 1a shows that the CodeRL with the CodeT5 model can achievesignificant performance gains, outperforming many pretrained LMs of much larger sizes.
Specifically,our approach achieved new SOTA results of 2 .
57 % pass@1 , 6 . 21 % pass@5 , and 19 .
36 % pass@1000 .
Table 1b shows that when evaluating on a subset of filtered code samples, our CodeRL+CodeT5 canachieve SOTA results of 7 . 83 % 1@k and 11 .
61 % 5@k .
Note that while CodeRL incurs additionalcomputation cost during inference with CS, our approach only requires much lower k to achievecomparable performance with other models.
Specifically, with k = 1000 only, our model performanceis as good as AlphaCode with much larger generation budget of k = 50000 .
4.3 Ablation Studies
In this section, for a fair comparison between variants of return estimates and learning objectives, wereport the results of pass@k where k = { 1 , 5 } with beam search decoding.
For larger k , we reportthe results with and without CS procedure.
[Table]
Impacts of Return Estimates.
Table 2 show the results of
CodeT5-770M trained by different approaches to estimate returns of code samples.
Overall,we report that the CodeRL objective with relative token-levelreturn estimates by our critic model (Model D) can achieve the best performance on pass@1 andpass@5 .
Secondly, we note that using absolute returns without a baseline (Model B) could lead to themost performance drop, as this approach heavily penalizes all incorrect samples (even though theymight still be better than a naive baseline).
Thirdly, without a critic model, simply assigning identicalrewards to all tokens in a code sample (Model A) is disadvantageous as these return estimates aretoo restrictive to be used as feedback signals for RL training.
Finally, we experimented with adistance-based critic which assumes that token values decay linearly from t = 1 to t =
T (Model C).
The lower performance suggests the benefit of training a critic network to compute the returns.
learning objectives
[Table]
Impacts of Learning Objectives.
Table 3 shows the resultswith different combinations of
L ce and L rl .
We experimentwith using only L rl and notethe problem of vanishing gradients during finetuning [Ranzato et al., 2016, Bahdanau et al.
Secondly, we note that byusing only L ce for further finetuning, despite improvement inlosses during training time, themodel performance indeed degrades during test time.
We expect these models are overfitting to the training data.
Interestingly,a naive approach of L ce with synthetic samples W s , all of which are treated as correct codes withr ( W s )
= 1 , still leads to some performance improvement with GPT-Neo on pass@5 (but not inother cases).
Finally, we found that using both L ce and L rl results in a more consistent performanceimprovement overall on pass@1 and pass@5 for the GPT-Neo and CodeT5 models.
[Table]
Impact of Critic Sampling .
Table 4 shows theablation results of critical sampling (CS) duringinference.
Overall, we found positive impact of
CS for improving pass@200 and pass@1000metrics.
Interestingly, we observe that CS doesnot provide a significant gain on the n@k metric.
Note that n @ k measures the solving rate amongthe subset F filtered from k samples.
As CSwill technically increase the size of this subset, the n@k metric will consider exponentially largernumber of options of n samples than before.
This will normalize n @ k by a larger pool of n candidateset, resulting in less impact of CodeRL on the results.
We recommend additional post-processingsteps such as candidate ranking [Cobbe et al., 2021] to improve the n@k performance.
Impacts of Pretraining Approaches for CodeT5.
Table 5 reports the results of CodeT5 withdifferent configurations of model sizes, pretraining data, and pretraining objectives.
[Table]
For a fair comparison, all modelsare only finetuned with L ce on
APPS.
As observed in prior work[Chen et al., 2021a, Austin et al.
,2021], scaling up the number ofmodel parameters or the size ofthe pretraining data can significantly improve model performance of downstream synthesis tasks.
We also find that enhancing the
[Table]
pretraining objectives with next token prediction (NTP) is vital for generation tasks, surpassing justmasked span prediction (MSP) from the original CodeT5.
4.4 Zero-shot Evaluation on MBPP Benchmark
[Table]
Finally, we test the zero-shot transfer ability of CodeRLon another smaller and simpler program synthesis benchmark MBPP [Austin et al., 2021].
Table 6 reports theresults of our CodeRL+CodeT5 on MBPP benchmarkcompared with finetuned GPT models of up to 137B size.
Our CodeRL+CodeT5 (ZS) was trained on APPS and thenevaluated on MBPP in a zero-shot setting.
We observethat CodeRL with CodeT5 of a much smaller model sizeyields surprisingly good zero-shot performance, setting anew SOTA result of 63.0% pass@80 over GPT-137B’s 61.4% pass@80 .
This validates the strongzero-shot transfer ability of CodeRL for unseen tasks.
More analysis is included in Supplementary.
Limitations and Broader Impacts
One limitation of our approach is the computation cost of training critic model to estimate returns inaddition to the original LM (actor network).
However, in practice, we found that training a good criticmodel does not require large-scale models to attain a decent performance.
For instance, a finetunedcritic model initialized from a pretrained GPT-2 (small) can achieve over 75% error predictionaccuracy on synthetic samples.
Program synthesis can lead to substantial positive social impacts, e.g., transforming future softwaredeveloping tools, increasing productivity of developers, and improving accessibility and quality ofprogramming courses.
Yet, some risks and bias issues are still worth considering before deployingsuch models at scale.
For example, training data from public github code repos may containvulnerabilities and the resulting synthesis models may generate programs with weak security measures[Hammond Pearce et al., 2021].
Conclusion
We present CodeRL, a novel framework for program synthesis, using deep reinforcement learningto improve pretrained LMs, by exploiting unit test signals in both training and inference stages.
Specifically, we introduce an actor-critic training approach to optimize pretrained LMs with densefeedback signals on synthetic code samples.
During inference, we propose a new generation procedurewith critical sampling, which enables the model to automatically regenerate programs based onfeedback from unit tests and critic scores.
We integrate CodeRL with the improved CodeT5-largemodel (770M) and achieve new SOTA results on both the APPS and MBPP benchmarks, surpassingthe prior SOTA by massive pretrained LMs of much larger model sizes.
Our comprehensive analysisshows that CodeRL achieved consistent improvement upon the conventional pretrained LMs for codegeneration tasks.
CodeRL is a general framework that integrates pretrained LMs and RL holisticallyfor program synthesis, and can be extended and improved in various ways.
For example, it canbe easily integrated with other better pretrained LMs and can be improved with more fine-grainedfeedback from the environment, such as feedback received from a static code analyzer.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 3 and
(b) Did you describe the limitations of your work?
[Yes] See Section 5.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See
Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
[Table]
(b) Did you include complete proofs of all theoretical results?
[N/A]
[Table]
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
[Table]
perimental results (either in the supplemental material or as a URL)?
[Yes] See theSupplementary.
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See the Supplementary.
[Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
[Table]
ments multiple times)?
[No] As it is very expensive to experiment with large-scalelanguage models, we did not try different random seeds due to the limitation of compu-
[Table]
tation resources.
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
[Table]
of GPUs, internal cluster, or cloud provider)?
[Yes] See the configurations in the
[Table]
Supplementary.
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] See the Supplementary.
[Table]
(b) Did you mention the license of the assets?
[Yes] See the Supplementary.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
We do not curate any new dataset in this paper.
We will release the code and models.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A] All datasets evaluated in our experiments are publicly available
[Table]
for use.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
The data we are using are code samplesfrom public programming competitions which do not include personally identifiable
[Table]
information or offensive content.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]