C ode BERTS core: Evaluating Code Generation with Pretrained Models of Code (2023)

Abstract: Since the rise of neural natural-language-to-code models (NL \rightarrow Code) that can generate long expressions and statements rather than a single next-token, one of the major problems has been reliably evaluating their generated output. In this paper, we propose CodeBERTScore: an evaluation metric for code generation, which builds on BERTScore (Zhang et al., 2020). Instead of encoding only the generated tokens as in BERTScore, CodeBERTScore also encodes the natural language input preceding the generated code, thus modeling the consistency between the generated code and its given natural language context as well. We perform an extensive evaluation of CodeBERTScore across four programming languages. We find that CodeBERTScore achieves a higher correlation with human preference and with functional correctness than all existing metrics. That is, generated code that receives a higher score by CodeBERTScore is more likely to be preferred by humans, as well as to function correctly when executed. We release five language-specific pretrained models to use with our publicly available code. Our language-specific models have been downloaded more than **1,000,000** times from the Huggingface Hub. Our code and data are available at https://github.com/neulab/code-bert-score

## introduction natural-language-to-code generation (@xmath0. llms have reached such a high @xmath1. this sharp rise in llms' usability was achieved thanks to their ability to accurately generate long completions, which span multiple tokens and even lines, rather than only a single next-token as in early models @xcite . nevertheless, evaluating and comparing different models has remained a challenging problem @xcite that requires an accurate and reliable evaluation metric for the quality of the models' generated outputs, and existing metrics are sub-optimal. existing evaluation approaches the most common evaluation metrics are token-matching methods such as bleu @xcite , adopted from natural language processing. these metrics are based on counting overlapping ngrams in the generated code and the reference code. crystalbleu @xcite extends bleu by ignoring the 500 most occurring n-grams, arguing that they are trivially shared between the prediction and the reference. nonetheless, both bleu and crystalbleu rely on the lexical exact match of tokens, which does not account for diversity in implementation, variable names, and code conventions. figure 1 shows an example: given the reference code in figure 1 (a), both bleu and crystalbleu prefer (rank higher) the non-equivalent code in figure 1 (b) over the functionally equivalent code in figure 1(c ). codebleu @xcite attempts to lower the requirement for a lexical exact match, by relying on data-flow and abstract syntax tree (ast) matching as well; nevertheless, valid generations may have different asts and data flow from the reference code, which may lead to low codebleu score even when the prediction is correct. further, partial predictions may be useful for a programmer, but accepting them may lead to partial code that does not parse, and thus cannot be fully evaluated by codebleu (for example, predicting the first line of a for loop, without the loop's body). execution-based evaluation attempts to address these problems by running tests on the generated code to verify its functional correctness @xcite @xcite @xcite . this provides a direct measure of the functionality of the generated code while being agnostic to diversity in implementation and style. however, execution-based evaluation requires datasets that are provided with hand-written test cases for each example, which is costly and labor-intensive to create; thus, only few such datasets exist. additionally, executing model-generated code is susceptible to security threats, and thus should be run in an isolated sandbox, which makes it technically cumbersome to work with iteratively. our approach in this work, we introduce code-bertscore, an evaluation metric for code generation, leveraging self-supervised pretrained models of code such as codebert @xcite , and adopting best practices bertscore @xcite . first, codebertscore encodes the generated code and the reference code independently with pretrained models, with the inclusion of natural language instructions or comments. then, we compute the cosine similarity between the encoded representations of each token in the generated code and each token in the reference code. finally, the best matching token vector pairs are used to compute precision and recall. code-bertscore allows comparing code pairs that are lexically different while taking into account the (1) programmatic-or natural-language-context, if such provided; the (2) contextual information of each token; and (3) implementation diversity. our approach is illustrated in figure 2 . example a concrete example is shown in figure 1 : while bleu and crystalbleu prefer (rank higher) the non-equivalent code in figure 1(b) given the reference code in figure 1 (a), code-bertscore prefers the code in figure 1 (c), which is functionally equivalent to the reference (figure 1(a) ). we note that in this example, the variable names are identical across all three code snippets. when the variable names of the reference are different than the candidate's, it is even harder for token-matching approaches such as bleu and crystalbleu to compare the reference with the candidates, while codebertscore can trivially match variable names according to their semantic similarity and their functional role in the code. contributions in summary, our main contributions are: (a) codebertscore: a self-supervised metric for @xmath2, based on bertscore, which leverages the benefits of pretrained models, while not requiring labeling or manually annotated data. (b) an extensive empirical evaluation across four programming languages, showing that codebertscore is more correlated with human preference and more correlated with execution correctness than all previous approaches including bleu, codebleu, and crystalbleu. (c) we pretrain and release five language-specific codebert models to use with our publicly available code, for java, python, c, c++, and javascript. as of the time of this submission, our models have been downloaded from the huggingface hub more than 1,000,000 times. ## experimental setup we evaluate codebertscore across multiple datasets and programming languages. we first show that codebertscore is more correlated with human preference than previous metrics, using human-rated solutions for the conala dataset @xcite . we then show that codebertscore is more correlated with functional correctness, using the humaneval dataset @xcite . we also show that codebertscore achieves a higher newly proposed distinguishability than other metrics (appendix f). finally, we analyze some of the design decisions and their implications. ## analysis we conducted a series of additional experiments to understand the importance of different design decisions, and to gain insights on applying code-bertscore to new datasets and scenarios. can we use codebertscore in a new language without a language-specific codebert? in all experiments in section 4, we used the language-specific model which we continued to pretrain on each language. but what if we wish to use codebertscore in a language in which we don't have a language-specific model? we compare the language-specific models to codebertbase in figure 4 . generally, codebert-base achieves close performance to a language-specific model. however, in most humaneval experiments and correlation metrics, using the languagespecific model is beneficial. these results show that language-specific models are often preferred if such models are available, but the codebertbase can still provide close performance even without language-specific pretraining. 0 2 4 6 8 10 12 0.25 0.3 0.35 0.4 0.45 layer java c++ javascript python figure 5: the average of kendall-tau and spearman on the development set of humaneval when using the embeddings from different layers. which transformer layer should we use? we further investigate the impact of using hidden states from different layers of the model -the layer which the vectors in equation (2) come from, in the computation of codebertscore. the results are shown in figure 5 : generally, the deeper the layer -the higher the average correlation between codebertscore and functional correctness, across all programming languages. however in almost all languages, performance reaches its maximum before the last layer, and decreases at the following layers. this suggests that higher layers encode the semantic information of each token more accurately, but the final layers may be more task-specific. these observations are consistent with @xcite , who found that lower layers in bert tend to process shallow informa-tion, while higher layers encode deeper semantic meaning in natural language. does encoding natural language context help? one major difference between codebertscore and bertscore is that codebertscore leverages the context for the generated code, such as the natural language instruction or intent that was given as input for generation. we find that using context increases the correlation, for example, the kendall-tau of codebertscore from 0.50 to 0.52. while this paper mainly focuses on natural language instructions, we believe that code-bertscore can thus benefit other programming scenarios as well, for example when generating code given the human-written comments, or generating code given the preceding code context. codebertscore allows soft matching of tokens the heatmaps in figure 6 show the similarity scores between tokens in codebertscore. for example, both shutil.rmtree and os.rmdir in figure 6 (a) delete a folder; codebertscore aligns each token to a respective token in the other expression, even though the two spans do not share many identical tokens. in figure 6 (b), both code snippets calculate a square root, where one uses math.sqrt(x) and the other uses x ** 0.5. an exact surfaceform-matching metric such as chrf would assign a low similarity score to this code pair, as they only share the token x. however, codebertscore assigns non-zero scores to each token with meaningful alignments, such as matching [sq,rt] with [_0,5], since a square root is the 0.5-th power. additionally, we study the robustness of code-bertscore to adversarial perturbations. we found that token-based metrics such as chrf are much more prone to matching trivial tokens rather than tokens that preserve the semantic meaning of the code. examples can be found in appendix e. additional discussion and experiments regarding the distinguishability of codebertscore are provided in appendix f. additional general examples are provided in appendix g. ## related work token-based metrics metrics such as bleu @xcite evaluate code generation by counting matching n-grams between generated and reference code. crystalbleu @xcite refines this approach by disregarding trivially shared n-grams, while rouge @xcite and meteor @xcite emphasize recall and balance of precision and recall respectively. however, these metrics, relying on exact lexical matches, often fail to capture semantically equivalent but lexically different code snippets. unlike these, codebertscore captures the wide, two-sided context of each token, which n-grams cannot capture. @xcite incorporates data-flow and abstract syntax tree (ast) matching, in addition to token-matching. however, valid code may not always align in asts and data-flows. additionally, partial code, although potentially useful, may not parse, thus cannot be fully evaluated by codebleu. further, as highlighted by subsequent studies @xcite , codebleu does not correlate well with execution accuracy. ## conclusion in this paper, we present codebertscore, a simple evaluation metric for code generation, which builds on bertscore @xcite , using pretrained language models of code, and leveraging the natural language context of the generated code. we perform an extensive evaluation across four programming languages which shows that codebertscore is more correlated with human preference than all prior metrics. further, we show that generated code that receives a higher score by codebertscore is more likely to function correctly when executed. finally, we release five programming language-specific pretrained models to use with our publicly available code. these models were downloaded more than 1,000,000 times from the huggingface hub. our code and data are available at https://github.com/ neulab/code-bert-score . 2022); the idea to use functional correctness as a meta-metric was born thanks to the discussion with her. we are also grateful to aryaz eghbali and michael pradel for the discussions about crystalbleu @xcite . this material is partly based on research sponsored in part by the air force research laboratory under agreement number fa8750-19-2-0200. the u.s. government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright notation thereon. the views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the air force research laboratory or the u.s. government. this project was also partially supported by a gift from aws ai.