Fault-Aware Neural Code Rankers
Abstract
Large language models (LLMs) have demonstrated an impressive ability to generate code for various programming tasks.
In many instances, LLMs can generate a correct program for a task when given numerous trials.
Consequently, a recent trend is to do large scale sampling of programs using a model and then ﬁltering/ranking the programs based on the program execution on a small number of known unit tests to select one candidate solution.
However, these approaches assume that the unit tests are given and assume the ability to safely execute the generated programs (which can do arbitrary dangerous operations such as ﬁle manipulations).
Both of the above assumptions are impractical in real-world software development.
In this paper, we propose C ODE R ANKER , a neural ranker that can predict the correctness of a sampled program without executing it.
Our C ODE R ANKER is fault-aware i.e., it is trained to predict different kinds of execution information such as predicting the exact compile/runtime error type (e.g., an IndexError or a TypeError).
We show that C ODE R ANKER can signiﬁcantly increase the pass@1 accuracy of various code generation models (including Codex [11], GPT-Neo, GPT-J) on APPS [25],
HumanEval [11] and MBPP [3] datasets.
Introduction
Large transformer-based language models (LLMs) have impressive capabilities [19, 7, 16], including the ability to generate code [11, 3, 28, 39, 33].
The task here is to take a natural language description or previous code context as the input and generate an entire program in a general-purpose programming language such as Python or C++.
Generating entire programs is a challenging task as it involves understanding the task, ﬁguring out how to accomplish it, and writing code without any syntax/runtime errors.
On harder programming problems such as coding competition problems, current models achieve very low accuracy especially if the inference-time sampling budget is low.
For example, on the APPS dataset [25], a state-of-the-art code generation model, Codex [11], achieves 4% accuracy if it is allowed to sample only one program per task (called pass@1), but achieves 24% accuracy if it is allowed 100 samples per task (called pass@100—at least one correct program in 100 samples) (see
Table 7).
This observation leads us to an important research problem on exploring approaches to rank the sampled programs to bridge the gap between the pass@1 and pass@100 performances.
Upon analyzing the sampled programs obtained using an LLM, we ﬁnd that several of them have syntax errors and runtime errors, and some of them execute to produce undesirable outputs.
Therefore, prior works [11, 28] focused on ranking the programs by executing them on a small set of unit tests (which are typically assumed to be given as part of the task description).
However, there are several caveats to this approach: First, even if a program passes the given tests, it could still fail on the unknown unit tests.
Second, there is a burden on the user to provide unit tests for every inference 36th Conference on Neural Information Processing Systems (NeurIPS 2022). [Figure]
task. Third, to execute the code, all the dependencies has to be installed properly.
This is especially problematic in scenarios where a user wants to get code suggestions in a project with multiple ﬁles and dependencies (such as using CoPilot in a VS Code environment [1]).
Even if these dependencies are satisﬁed, many realistic programming scenarios involve incomplete code under active development where execution is just infeasible.
Finally, the code generated by a LLM could potentially have security risks (such as deleting ﬁles on the disk) and hence, executing such a code needs heavy-weight isolation mechanisms such as a sandbox environment.
To alleviate the above issues with relying on executing code, some of the recent works [18, 36] proposed to use a neural-network based ranker 1 for ranking programs sampled from a LLM.
A ranker model is essentially a classiﬁer that takes as input a task description and a sampled program and predicts the probability that the program is correct with respect to the task description.
When given multiple programs (obtained by sampling), a ranker re-orders them in the decreasing order of the predicted probability of the program being correct.
The ranker is essentially trying to emulate executing a program on some unit tests without actually executing the code during inference.
The training data is obtained by executing both correct and wrong programs sampled from the code generation model itself.
Thus, we only need unit tests and the execution ability during the dataset creation step instead of during inference as in prior approaches.
While previous approaches target math problems to make the learning problem tractable, in this paper, we design C ODE R ANKER for more complex general programming tasks in Python.
A sampled Python program can fail in many different ways.
For example, a program when executed on a unit test can result in a compile/runtime error and can produce a wide variety of outputs such as integers, strings, lists and dictionaries that can produce a type mismatch.
In contrast in the math domain, there is less scope for compile/runtime errors and the output is usually just a number.
Our C ODE R ANKER approach is based on the idea that a neural ranker trained to distinguish between the various failure modes can have a better understanding of the program and the task, and hence can do better at ranking programs.
Thus, we design fault-aware C ODE R ANKER and we investigate its impact on the ranking performance.
Each fault-aware ranker is a classiﬁer trained to predict one/two multi-class labels that are extracted from the rich information obtained by executing the programs.
We used the APPS dataset to ﬁnetune/train the code generation and the C ODE R ANKER models.
On this dataset, we showed that C ODE R ANKER improved the pass@1 performance of Codex (used in a few-shot manner) from 26% to 39.6% on the validation set and from 3.8% to 4.5% on the test set.
We additionally found our rankers are transferable to a different dataset without any additional training.
On the HumanEval dataset, we improved Codex’s pass@1 from 26% to 32% and on the
MBPP dataset, we improved from 36% to 42%.
We found similar performance boosts with other code generation models such as GPT-J and GPT-Neo.
Compared with a naïve binary classiﬁer-based ranker, our fault aware C ODE R ANKER achieves better ranking performance.
Finally, we investigated the effect of mixing ranker datasets from multiple code generation models which gives us an extra boost in the performance.
Preliminaries 2.1 Code generation
Task:
A code generation task G is a prompt, represented as a sequence of tokens, that speciﬁes the task at hand.
G is usually a combination of natural language, input-output examples, and starter
[Table] code. A solution S to a code generation task is a sequence of tokens that together form a program to solve the given task.
Existing datasets additionally contain a set of input-output pairs that are used to test the correctness of a generated program.
Figure 1 gives an example code generation task, the corresponding reference solution, and the unit tests taken from the APPS dataset [25].
Models: There are several existing pre-trained LLMs in the literature that are suitable to generating code either in a few-shot manner or after ﬁnetuning.
A code generation model F provides a way for us to sample programs for a given task G as S i ∼ P F ( S | G ) where P F denotes the probability distribution induced by the code generation model F .
In this paper, we study four different code generation models—(i)
Codex, (ii) GPT-J (6B), (iii)
GPTNeo 1.3B and (iv) GPT-Neo 125M (Table 1). Codex is the largest state-of-the-art code generation model that is publicly available for querying through an API and has shown impressive performance in code generation [11].
It is built on top of a GPT-3 language model architecture and is trained on 180 GB of GitHub data.
GPT-Neo and GPT-J are open-sourced models with the number of parameters ranging from 125M to 6B. These models are pre-trained on the Pile dataset (800 GB of natural language corpus with 8% of GitHub data).
Since, these models are open-sourced, it is possible to ﬁnetune these models on a downstream dataset such as the APPS dataset.
In addition to the above models, there are other code-generation models such as AlphaCode [28] and Google’s model [3].
While our approach is applicable to any of these models, in this study, we validate the effectiveness of C ODE R ANKER on a set of state-of-the-art code generation models that are publicly available, for reproducibility.
Metrics: Code generation models are evaluated based on functional correctness rather than exact/- fuzzy match to a reference program.
This is because match-based metrics are unable to account for the large and complex space of programs functionally equivalent to a reference program.
Functional correctness is estimated by checking whether a sampled program passes a set of unit tests.
Prior approaches evaluate functional correctness using the pass@k metric; given k generated program samples per task, a task is considered solved if any of the samples passes the unit tests.
The pass@k metric measures the total fraction of tasks solved.
Additionally, we deﬁne the exec@k metric which computes the fraction of the tasks for which there exists at least one program in the k samples that executes without any compile/runtime errors, i.e., produces a non error value for each of the test inputs, but may or may not match the desired output.
2.2 Code ranking
Given n sampled programs using a code generation model, S 1 , · · · , S n ∼ P F ( S | G ) , the goal of C ODE R ANKER is to ﬁnd an ordering of the programs S o 1 , · · · , S o n such that to compute the ranked pass@k for k ≤ n , a problem is considered solved if any program in the set { S o 1 , · · · , S o k } passes the unit tests.
A C ODE R ANKER model R takes as input a code generation task G and a sampled program S i and outputs a score
s i .
The ranked ordering of the sampled programs is given by
S o 1 , · · · , S o n such that s o 1 > s o 2 > · · · > s o n .
Fault-Aware Neural Code Ranker
A C ODE R ANKER model is a classiﬁer that is trained to classify a pair (cid:104) G, S i (cid:105) as CORRECT or not, where CORRECT means that S i satisﬁes the task G with respect to its unit tests.
The score s i is computed as s i
= P R ( CORRECT | G, S i ) where P R is the probability according to the ranker model R .
This probability is extracted using the real values from the last layer before the SoftMax layer.
[Table]
[Table]
3.1 Code Ranker Dataset
To train a C ODE R ANKER model, we need a dataset that has both CORRECT and WRONG (i.e., not correct) programs.
To collect these programs, we use the code-generation models to sample n =programs for each task in the training dataset.
We then execute the sampled programs on their corresponding unit tests to generate the classiﬁcation labels.
Following the observations from [18], one has to be careful to not over-train the base code generation models to ensure diversity in the sampled programs.
Hence, we only ﬁnetune the base code generation models for a maximum ofepochs and chose a checkpoint that results in the lowest validation loss (this does not apply to the Codex model, which is used in a few-shot manner).
Table 2 shows the distribution of the ranker’s datasets obtained by using the 4 different code generation models for the tasks in the APPS dataset 2 .
As one can expect, the ranker dataset is highly imbalanced with about 5X to 40X more WRONG data points than CORRECT data points and this ratio is higher for smaller models such GPT-Neo models.
Fault-aware ranker dataset: A straightforward ranker is one that is trained to predict a binary label
CORRECT or WRONG .
However, a program fails for various reasons and knowing why a program might fail is crucial to predicting whether a program is CORRECT or not.
Therefore, we designed a fault-aware ranker dataset.
When we execute a program on a set of unit tests, the compiler message is more than just a single bit of information.
In fact, when the unit test fails, we know if it failed because of a compile/runtime error (which we call an execution error ) or because the program produced a wrong output for a particular input (which we call an intent error ).
Table 2 also shows the distribution of intent errors and execution errors in the ranker datasets.
An interesting observation is that the percentage of execution errors decreases for larger code generation models.
It is possible to further break down the WRONG datapoints.
Within the execution error class, the compiler message tells us exactly the type of the execution error (such as an IndexError or a TypeError or a TimeOutError) and the line in the program that caused this error.
By parsing the error message from the Python compiler, we derived 10 most frequent classes of execution errors as shown in
Table 4 and Figure 2.
Similarly, for the intent error class, we know how the generated output differs from the expected output (such as wrong type or wrong length of the array output). We manually
[Table] [Table] designed 9 most frequent classes of intent errors by looking at various expected and generated outputs from the training examples (see Table 5 and Figure 3).
These ﬁne-grained execution-based labels constitute our fault-aware dataset.
Table 3 shows a few data entries with all the labels.
Figure 6 in the Appendix shows the distribution of the various classes of execution errors and intent errors for the ranker dataset obtained using the Codex model [Table]
3.2 Code Ranker Tasks
We now describe the different classiﬁcation tasks derived from the above dataset.
For each classiﬁcation task, the input is a pair of code generation task speciﬁcation G and a generated program S i and the output is one or multiple labels where each label belongs to pre-determined set of classes.
We describe the different labels/classes of the various tasks below.
These tasks are designed to explore the trade-offs between having abstract classes of failures versus having ﬁne-grained classes of failures.
Binary (B): The output is a single binary label with two classes { CORRECT , WRONG }.
Ternary (T): The ternary task splits the WRONG class into intent error and execution error classes: this forms a three-class classiﬁcation task with output labels { CORRECT , intent error, execution error}
Intent Error aware (I): The intent error aware task splits the intent error class in the ternary task into its 9 different sub-classes, thus has a total of 11 classes for the output label.
Execution Error aware (E): The execution error aware task is similar to the intent error aware task but instead of splitting the intent error class, we now split the execution error class into its 10 different sub-classes, thus has a total of 12 classes for the output label.
Execution Error + Error Line aware (E+L):
This is a multi-class and multi-label classiﬁcation task which combines two classiﬁcation tasks.
The ﬁrst one is the execution error aware task described above.
The second task is to predict the line of the code that corresponds to an execution error.
The labels for the error line number also includes − 1 to represent there is no execution error.
3.3 Code Ranker Models
We implemented our fault aware C ODE R ANKER models by ﬁnetuning the pretrained CodeBERT [21] model.
CodeBERT:
It is a state-of-the-art pretrained BERT-style code-understanding model trained on the
CodeSearchNet dataset [26] using a combination of masked language modeling and replaced token detection objectives [17].
It takes as input a concatenation of two segments with a special separator token, namely [ CLS ] , w 1 , w 2 , ..w n , [ SEP ] , c 1 , c 2 , ..., c m , [ EOS ] .
Usually, one segment is a natural language text, and another is a code.
[ CLS ] is a special token, whose ﬁnal hidden representation can be treated as the aggregated sequence representation for classiﬁcation or ranking downstream tasks.
Adding a classiﬁcation head: We add a classiﬁcation head on top of a base CodeBERT model by connecting a linear layer and a softmax layer to the hidden representation of the [ CLS ] special token.
Let C ∈ R H be the ﬁnal hidden vector corresponding to the [ CLS ] token and let W ∈ R K × H be the [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
weights of the newly added classiﬁcation layer where K is the number of classes in the classiﬁcation task.
The logits for the classiﬁcation output are computed as softmax ( CW (cid:62) ) and we use a standard cross-entropy loss for ﬁnetuning all the weights.
Adding a line prediction head: To predict the line corresponding to an execution error, we introduce an error line vector S ∈ R H during ﬁne-tuning.
The probability of a newline (“\n”) token i being the erroneous line is computed as a dot product between T i and S followed by a softmax over all of the newline tokens in the code, i.e., P i = [Table]
j e STj where T i is the ﬁnal hidden vector corresponding to the i th newline token.
We include a newline token at the beginning and the end of the input to indicate the case where there is no erroneous line in the code (i.e., code does not result in an execution error) and to indicate the case where the erroneous line is beyond what is encoded in the input (this occurs if the task+code context cannot ﬁt within the 512 token limit of CodeBERT), respectively.
Evaluation
We next evaluate our C ODE R ANKER approach.
We investigate (1) how fault-aware C ODE R ANKER s can improve various code generation models on various code datasets, (2) the impact of the different Code gen. model [Table] ranker tasks, and (3) the effect of mixing ranker datasets generated by different code generation models.
4.1 Experiment Setup
Code generation datasets: We consider three existing code generation datasets for our evaluation: (1) APPS [25]: a collection of 5000 training and 5000 test tasks collected from coding competitions and interview problems, (2) HumanEval
[11]: a set of 164 test tasks, and (3) MBPP [3]: a set ofmostly basic python programming tasks with 474 training problems and 500 test problems.
In our experiments, we only use the APPS dataset for ﬁnetuning the code generation models and the C ODE R ANKER models (since it is the largest dataset).
But we evaluate these models on all three sets of test tasks.
The APPS dataset does not come with a validation dataset, so we used a set of 600 tasks from the original training dataset for validation; these are, then, excluded from the training dataset.
Since we are interested in explicitly evaluating the ability of a ranker to distinguish CORRECT code from WRONG code, we chose our validation set to only include problems for which a Codex model(in a few-shot manner) can generate at least one correct program in its 100 samples.
To facilitate the transfer of GPT-J and GPT-Neo ﬁnetuned models on the HumanEval and the MBPP datasets, we perform a minor programmatic transformation of the task descriptions to match the APPS style.
Metrics: We use the pass@1, pass@5, exec@1, ranked pass@1, ranked pass@5, and ranked exec@ metrics (higher values are better).
See Section 2 for their deﬁnitions.
We also show the pass@ metric to illustrate the maximum possible value for the pass@k and ranked pass@k metrics.
These metrics are measured using an unbiased estimator from 100 samples as proposed by [11].
Training setup and hyper-parameters: We ﬁnetuned GPT-J and GPT-Neo code generation models on the APPS training dataset for 2 epochs with a batch size of 256 and a learning rate of 1e-5, and chose the checkpoint that has the lowest validation loss.
For inference, we used temperature sampling with T = 0 .
8 for Codex model and T = 0 .
9 for the GPT-J and GPT-Neo models unless speciﬁed otherwise.
We chose these temperatures to maximize diversity in the 100 samples, but we also conduct an ablation with lower temperatures in Table 11.
For each program, we sample 512 new tokens and truncate the generated program by a special stop sequence that we used in the few-shot/ﬁnetuning prompts.
We ﬁnetuned the C ODE R ANKER models for 30 epochs with a batch size of 512 and a learning rate of 1e-4, and chose the checkpoint that results in the best ranked [-pass@
 [Table] [Table] [Table]-]pass@1 metric on the validation dataset.
We used class weights to balance the different classes while training the rankers.
All experiments are conducted on V100-32GB GPUs.
Notation : We use the notation R YD X to denote a C ODE R ANKER model trained on a dataset obtained using the code generation model X from Table 2 and on one of the ﬁve ranker tasks Y from Section 3.2.
4.2 Main Results: C ODE R ANKER improves code generation models
APPS validation dataset: First, we analyze the results on the APPS validation dataset oftasks.
Table 6 shows the performance on various metrics for the 4 different code generation models.
These results use the best C ODE R ANKER model for each code generation model which is shown in Table 10.
From Table 6, we ﬁnd that C ODE R ANKER s improve all the metrics for all the code generation models despite their different sizes.
The pass@1 performance increases by 5.1% to 13.6% with C ODE R ANKER and the models can solve about 30 to 80 more tasks when it has to select only one program from the 100 samples.
Another interesting observation is that a GPT Neo 125M model when combined with C ODE R ANKER (another 125M model) beats a GPT-J model (with 50X more parameters).
These results show the effectiveness of C ODE R ANKER on improving the code generation models.
APPS test dataset: Our results on the APPS test dataset of 5000 tasks is shown in Table 7.
The test problems are harder than the ones on the validation set, which we can see by the smaller pass@ [Table] [Table] and pass@1 numbers.
Hence, the improvement from C ODE R ANKER is smaller in scale, but it is still a signiﬁcant improvement; Codex’s pass@1 increases from 3.8% to 4.7% (35 additional problems).
HumanEval and MBPP datasets: We measure the transfer ability of C ODE R ANKER on two different datasets–HumanEval (results in Table 8) and MBPP (results in Table 9).
We can again see that C ODE R ANKER s improve the performance of all code generation models on all metrics for both datasets (one exception is pass@5 for GPT-J on MBPP).
Codex model’s pass@1 performance increases by 6% on both datasets.
These results show the ability of C ODE R ANKER to transfer out-ofdistribution and also showcases that the errors made by the code generation models on different tasks are universal.
Exec@1 vs pass@1:
In all the above results, the improvements in the exec@1 metric are higher than the improvements in the pass@1 metric; this shows that C ODE R ANKER is better at identifying execution errors in code than intent errors, which is expected since executing code is shown to be an inherently hard task for language models [3, 34].
4.3 Ablations
Effect of sampling temperature: On the HumanEval and the MBPP datasets, we additionally experiment with different temperatures for sampling the 100 programs (see Table 11).
As expected, we noticed that pass@100 decreases with lower temperatures, but pass@1 increases. We found that
C ODE R ANKER further increases the pass@1 performance for 3 out of the 4 setups and achieved 42.7% ranked pass@1 on the HumanEval dataset—the best known result so far on this dataset 4 [16].
On the MBPP dataset, under a low-temperature sampling setup, we found that C ODE R ANKER slightly decreases the pass@1 performance—this we attribute to the small difference between the pass@1 and the pass@100 metrics, which makes it hard for a learned ranker to beat a random ranking scheme.
Analyzing different ranker tasks: Figure 4 shows 4 training curves; one for each code generation model, X , showcasing the validation ranked pass@1 curves for the model X when combined with the 5 different rankers R ∗ D X .
Results in a tabular format can be found in the Appendix.
From the curves, we can notice that for larger models such as Codex and GPT-J, the rankers trained on the ternary classiﬁcation task R T perform the best, and for smaller models such as GPT-Neo 1.3B and GPT-Neo 125M, the ranker trained on the intent-aware classiﬁcation task R I and the ranker trained on the execution-aware + error line classiﬁcation task R E + L perform the best, respectively.
We can also notice that the binary rankers R B perform signiﬁcantly worse especially with smaller models.
These results suggest that when training rankers on a dataset that has more WRONG code, harder classiﬁcation tasks act as better regularizes.
Figure 7 in the Appendix shows similar training curves for the ranked exec@1 metric; here, we can see that R E and R E + L always achieves best ranked exec@1 because they are trained to identify execution errors and R B and R
I have the worst performance.
[Figure]
[Table]
[Figure]
Analyzing different ranker datasets: In this experiment [Figure] we measure the impact of using the rankers trained on data from one code generation model on another code generation model and the effect of mixing these ranker datasets.
We have 4 different code generation models in this experiment and hence, 4 different ranker datasets.
We analyze two different mixed datasets—(i) mixed-small that randomly samples 25% of the above ranker datasets and combines them, and (ii) mixed-large that combines all of the 4 ranker datasets into one; the former represents a dataset that is roughly the same size of the individual ranker datasets for a fair comparison, while the latter makes use of all the available data.
Figure 5 shows the ranked pass@1 of all the 4x6 combinations in the form of a heat map.
Each column has been normalized such that the values for the same model is 1 (i.e. the diagonal values are 1).
From the ﬁgure, among the individual ranker datasets, we notice that the rankers trained with the data from the same code generation model are the best and that the rankers trained with data from code generation models with a large size difference are the worst.
Finally, the rankers trained on mixed-small dataset perform slightly worse (except for GPT-J code gen. model) than using the same model dataset, but the rankers trained on mixed-large dataset have the overall best performance. These results suggest that while it is usually better to use the same model for generating the ranking dataset, we can potentially use smaller models (less expensive models) to augment the ranker dataset to further improve the performance.
Additional qualitative/quantitative analysis of the C ODE R ANKER approach can be found in the Appendix.
Related works
Code generation tasks/models: There are several code generation models explored in the literature, ranging from decoder-only architectures [11, 3, 39, 6, 22] to encoder-decoder architectures [28, 33, 2, 38] of various sizes.
Similarly, there are several task datasets from multiple domains, including math-word-problems [3, 18], Jupyter notebook cell generation [10], common programming tasks [3, 11, 35] and competition level programming problems [28, 25].
In this paper, we evaluate C ODE R ANKER ’s ability to improve the performance of four decoder-based code generation models on three programming datasets.
Our approach is agnostic to code generation model architectures as long as they generate outputs by sampling and our approach can be easily extended to different domains.
Code understanding tasks/models: Besides code generation tasks, there are several code understanding tasks including code search [4, 8, 23, 26], clone detection [37, 30], code summarization [26], code translation [13, 27, 32] and defect detection [9, 5, 15, 40, 5].
Encoder-only code understanding models [29, 21, 2, 38, 24] are developed for these tasks.
Our C ODE R ANKER task can be viewed as a new code understanding task, and our ranker model is ﬁnetuned from CodeBERT [21].
The defect detection datasets are closely related to ours; the main difference is that prior work focused on ﬁnding vulnerability in human-written code, while we focus on detecting errors in model-generated code.
Filtering/ranking for code generation models: Previous works such as [11, 28] use execution to prune code completions during inference.
While [11] only uses unit tests provided as part of the task, [28] additionally uses a neural model to generate inputs for the unit tests and uses execution to ﬁlter out programs that produce the same outputs on those inputs.
However, these work requires executing potentially vulnerable code for every inference task.
Our approach bypasses execution at inference time with the neural ranker.
Similar to our work, [18, 36] propose a neural-network based ranker/veriﬁer for code generation models; the main difference is the domain (general purpose programming tasks in our case versus math problems in prior work) and we train our neural rankers to learn why/how a code fails rather than just predicting a binary label.
Moreover, [18, 36] use generative models as their ranker base; in our work, we show beneﬁts of rankers using a simple encoder-only model with only 125M parameters ( [18] uses models with atleast 3B parameters) (see
Section A.2.3 in the Appendix to see our ablation on different ranker architectures). Using execution/static analysis in other contexts: There has been other works on using execution to guide code generation by conditioning the generation on a representation of the program states [12, 20].
There are also attempts to replace the execution process with a neural model in these cases [14, 34].
Another relevant work is [31], which improves the code generation models by using static analysis to augment the model’s input and has shown to drastically reduce the number of execution errors made by the generated programs.
This paper takes an alternate approach by using a neural ranker model to prune out wrong programs at inference time.
Conclusions and Future Directions
We presented C ODE R ANKER that ranks programs generated by a code generation model without explicitly executing the programs.
Our rankers are fault-aware i.e., trained to predict the ﬁne-grained classes of failure modes and we showed the effectiveness of the fault-aware rankers in improving pass@k and exec@k metrics for various code generation models and tasks.
One of the main limitations is that the C ODE R ANKER approach is not sound, i.e., a ranker can classify a correct program as wrong and vice-versa.
Additionally, we incur extra inference time to get better performance, because we now have to generate n >> k programs to ﬁlter k programs to show to the user.
Our current approach also relies on sampling full programs from the code generation model before using C ODE R ANKER .
In the future, we want to investigate ranking/classifying partial programs, which can in-turn reduce the inference time by pruning out wrong programs early.
Another future direction for our work is to investigate other ranker model architectures such as those that can better levarage the code structure and exploring other ranker tasks such as generating the full error message.
Finally, it will be interesting to investigate the transferability of our C ODE R ANKER approach to generic programming tasks (rather than just competition programming).
A main challenge. here, is the lack of a generic programming dataset which needs to collected by scraping public sources such as GitHub. Our preliminary experiments on such a generic programming dataset show that existing code generation models usually produce more programs with execution errors rather than programs with intent errors .
This observation combined with our results in this paper that show C ODE R ANKER is usually better at identifying execution errors than intent errors, we are very optimistic that a fault aware C ODE R ANKER approach would also improve the code generation for generic programming tasks. Acknowledgements: We thank Todd Mytkowicz, Piali Choudhury, Rahee Gosh Peshwaria, Curtis von Veh, and Xiaodong Liu for helpful discussions on this work. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] In Section(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Our approach increases inference time as mentioned in Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code and data is released on GitHub https://github.com/microsoft/CodeRanker .
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
In Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] It is very expensive to run the experiments multiple times.
But, we tried multiple runs for one of our rankers in early experiments, and we did not see much variance in the results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We report the resources used in Section 4.1 and in Section A.2.3. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] The assets are publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Table 12 and Table 13 show additional examples for each execution error and intent error class. Figure 6 shows the distribution of the various classes of execution and intent errors for the ranker dataset generated by the Codex model using the APPS training dataset. A.2 Additional results A.2.1 Ablations on different ranker tasks
Table 14 and Table 15 show the various ranked metrics for rankers trained on different fault-aware tasks when using the Codex and GPT-Neo 125M code generation models respectively. Figureshows 4 training curves, X , showcasing the validation ranked exec@1 curves for the model X when combined with 5 different rankers R ∗ D X . [Table] Function not found
[Table]
A.2.2 Ablations on different ranker datasets
Table 16 shows the ablation results when training rankers on datasets collected from code generation models.
These results show the improvements of rankers on the Codex code generation model.
We can see that a ranker trained on completions from the Codex model is better than a ranker trained on completions from other models.
We can also see that a ranker trained on the mixed-large dataset (which use all completions from all models) beats the ranker trained on Codex dataset by additional 2.7%.
Example
[Table]
[Table]
[Table]
A.2.3 Alternate ranker architectures
We experimented with a GPT-Neo 125M decoder-based model for the ranker and found it to be performing suboptimal to a CodeBERT-based ranker (even though both models are roughly the same size).
The GPT-Neo 125M ranker trained on the Codex ranker dataset ( R TD codex ) achieved 36.8% ranked pass@1 when combined with the Codex model while the CodeBERT-based ranker achieved 39.6%.
[Table]
[Table]
We further conducted an experiment with a GPT-Neo 1.3B model for the ranker.
It took almostdays to complete 7 epochs of training with 16 V100 GPUs running in parallel.
And still, it didn’t reach the performance achieved by the CodeBERT model.
GPT Neo 1.3B ranker model achieved ranked pass@1 of 38.6% while CodeBERT ranker achieved 39.6%.
We stopped GPT Neo 1.3B ranker training at 7 epochs because the training became unstable at that point.
On the other hand, a
CodeBERT based ranker took only 12 hours to train for 30 epochs (with 16 GPUs)
[Table]
Table 17 shows the ranked pass@1 and ranked exec@1 performance of these various ranker architectures and Figure 8 shows the smoothed training curves for these three rankers.
We believe that a ranker task is a code understanding task, hence starting with pre-trained encoders such as CodeBERT are most effective compared to starting with a pre-trained encode-decoder or decoder only code-generation models.
We leave the full study about the impact of various ranker architectures to future work.
We also do not anticipate that just bigger ranker models would ﬁll in the gap between current ranked pass@1 and pass@100 since a ranker needs to implicitly learn to execute which is a hard task.
For that, we might need new architectures for capturing code structure and execution.
For this future exploration, we hope that our ranker tasks will serve as good benchmarks for evaluating such architectures.
[Table]
A.2.4 Impact of example unit tests in the prompt
Following prior works, our experiments so far include a small number of unit tests in the task description of the problem (as seen in Figure 1) so that we can directly compare our results with prior works.
Although our C ODE R ANKER approach does not explicitly use these unit tests (since we do not assume that an execution environment is given), our rankers can use the given unit tests as part of their language/code comprehension ability to better solve the tasks.
So, in this section, we conduct an ablation study to measure the impact of the presence of these unit tests in the prompt on the ranking ability of C ODE R ANKER .
We created a best-effort version of the APPS training/validation dataset
[Figure]
that does not include the unit tests in the prompts and we trained a new ranker on this dataset.
This new ranker only performs slightly worse than the ranker that is trained with the unit tests in the prompts, but it still signiﬁcantly outperforms the approach without any ranker (see Table 18).
Codex + Ternary Ranker pass@ exec@
Prompts with example unit tests 39.
Prompts without example unit tests 38.
Without ranker 26.
A.2.5 Comparison to ranking using the code gen. model’s predicted probabilities
In this ablation, we compare the impact of ranking using a separate ranker model versus ranking using the code gen. model’s predicted probabilities for the various generated programs.
The results for the GPT-Neo 125M code gen. model can be found in Table 19; we can see that a separate
C ODE R ANKER produces the best ranking effect and in fact, using the intrinsic code gen. model’s probabilities to rank results in a slightly worse performance compared to a no ranking approach.
pass@ pass@ pass@ exec@
GPT-Neo 125M 23.
41.1 + ranking using generation probability
[Table]
A.2.6 Qualitative analysis of C ODE R ANKER
In Table
[Table]
we show various programs generated by the ﬁne-tuned GPT-J model, the actual error message from the Python compiler upon executing these programs
[Table]
and the various labels generated by the different C ODE R ANKER s trained on various ranker tasks
[Table]
Generated code
Actual error message
Predicted
[Table]
Expected True, but got False
[Table]
Passed all tests
[Table]
NameError(base is not deﬁned) at Line
[Table]
c
[Table]
[Equation]
[Table]
[Equation]
else :
[Equation]
[Table]
Got [[’7’], [’13’]], but expected [[’1’], [’3’]]
[Table]
TypeError(sequence item0:expectedstr instance, int found) at
Line
[Table]