ICE -Score: Instructing Large Language Models to Evaluate Code (2024)

Abstract: Recent advancements in the field of natural language generation have facilitated the use of large language models to assess the quality of generated text. Although these models have shown promising results in tasks such as machine translation and summarization, their applicability in code intelligence tasks remains limited without human involvement. The complexity of programming concepts required for such tasks makes it difficult to develop evaluation metrics that align with human judgment. Token-matching-based metrics, such as BLEU, have demonstrated weak correlations with human practitioners in code intelligence tasks. Moreover, utilizing human-written test suites to evaluate functional correctness can be challenging in domains with low resources. To overcome these obstacles, we propose ICE-Score, a new evaluation metric via instructing large language models (LLMs) for code assessments. Our metric addresses the limitations of existing approaches by achieving superior correlations with functional correctness and human preferences, without the need for test oracles or references. We evaluate the efficacy of our metric on two different aspects ( human preference and execution success ) and four programming languages. Our results demonstrate that our metric surpasses state-of-the-art metrics for code generation, delivering high levels of accuracy and consistency across various programming languages and tasks. We also make our evaluation metric and datasets available to the public, encouraging further research in evaluating code intelligence tasks.

## introduction natural language generation (nlg) systems have seen significant progress in developing large language models (llms). these models have shown great promise in generating high-quality and diverse texts that can be difficult to distinguish from 1 https://github.com/terryyz/ice-score human-written texts @xcite . however, evaluating the quality of nlg systems remains a challenging task, primarily due to the limitations of traditional evaluation metrics. tokenmatching-based metrics, such as bleu @xcite and rouge @xcite , have been widely used to evaluate nlg systems but have demonstrated poor correlation with human judgment and a lack of ability to capture semantic meanings @xcite . furthermore, these metrics require reference output, which can be challenging to obtain for new tasks and low-resource domains @xcite . in recent years, the use of llms as referencefree evaluators for natural language generation (nlg) tasks has gained attention among researchers. this approach is strongly aligned with human preferences, even when reference texts are unavailable @xcite . the underlying assumption behind this approach is that llms possess a profound understanding of humangenerated text and task instructions, enabling them to evaluate various nlg tasks through prompts. the exceptional performance of llms in contextual understanding and natural language generation, as evidenced by studies @xcite , further supports this assumption. moreover, llms trained on both textual and code-based data have showcased remarkable capabilities in diverse downstream tasks related to source code, including code generation @xcite @xcite . while a performance gap still exists between llms and human developers in coderelated tasks, recent research has illustrated that llms can be enhanced to handle various source code tasks with appropriate guidance @xcite . this indicates the significant potential of llms in comprehending and working with source code. code evaluation presents unique challenges, requiring a deeper understanding of programming concepts and more complex syntax than natural language generation @xcite . traditional reference-based evaluation metrics for code generation, such as bleu @xcite , rouge @xcite , and chrf @xcite , rely on token matching to assess performance automatically. however, these metrics have demonstrated poor correlation with human evaluation @xcite since they often underestimate the variety of outputs with the same semantic logic. while some studies have incorporated programming features to improve these metrics, they have shown limited gains and poor correlation with functional correctness @xcite . alternatively, researchers have proposed using well-designed test suites to objectively evaluate code generation performance at the function level @xcite @xcite . however, developing these test suites requires programming expertise, which can be impractical and costly in low-resource scenarios. additionally, executing model-generated code poses a security risk and must be run in an isolated sandbox, which is technically cumbersome. more recently, codebertscore @xcite , a neural-model-based evaluation metric, has been proposed, showing a higher correlation with functional correctness and human preferences by capturing the semantic information of reference code and generated code. however, codebertscore still relies on high-quality references that can be difficult and expensive to obtain. moreover, the limited performance of the code-bert @xcite backbone suggests that it has not yet reached a human-level understanding of source code, limiting the effectiveness of codebertscore. therefore, more advanced evaluation metrics are needed so that they can better capture the complex syntax and semantics of code intelligence tasks. to address these challenges, we propose a novel evaluation metric based on llms trained on both text and code, shown in figure 1 . specifically, we instruct llms to perform human-like multidimensional code evaluation, where the metric is denoted as ice-score. our metric leverages the recent nlg metric, g-eval @xcite , but achieves superior correlations with subjective human preferences and objective functional correctness, both at the example and corpus levels. different from g-eval, ice-score only relies on assessment criteria and evaluation step template, without the need for instruction generation and weighted scoring function. based on our extensive evaluation, we have summarized our contributions as follows: ## method our evaluation metric ice-score, inspired by g-eval @xcite , consists of two main components: 1) task definition, evaluation criteria, and detailed evaluation steps, and 2) a given problem and generated code snippet for evaluation. different from g-eval, we only require the input of evaluation criteria and template-based evaluation steps, without the need for generation from llms. in addition, as we set the model temperature to 0, our evaluation metric no longer needs a weighted scoring function after iterative score generation. these two differences suggest that ice-score is more cost-friendly and efficient. ## experiment setup we evaluate the effectiveness of ice-score using gpt-3.5 (gpt-3.5-turbo foot_1 ) as the backbone across multiple datasets and programming languages. we conduct two experiments to investigate the correlation between ice-score and human preference and functional correctness, respectively. we compare the performance of llmbased evaluations against 7 predominant automatic evaluation metrics, including the state-of-the-art codebertscore @xcite . to measure the correlation with human preference, we use the conala dataset @xcite and corresponding human annotation on the generated code from various models trained on the dataset @xcite . to measure the correlation with functional correctness, we use the humaneval-x dataset @xcite . we do not consider distinguishability as an evaluation option, as prior work @xcite has shown it to be an unreliable meta-metric that cannot substitute for execution-based or human-based ratings. ## results human-based usefulness metric java c++ python javascript average τ r s τ r s τ r s τ r s τ r s bleu .267 .326 .225 .276 .281 .344 .220 .270 .248 .304 codebleu .293 .359 .212 .260 .303 .371 .315 .385 .281 .343 chrf .290 .355 .266 .325 .328 .402 .279 .342 .291 .356 rouge-l .280 .342 .234 .286 .296 .363 .216 .264 .256 .314 meteor .318 .389 .260 .319 .349 .427 .311 .380 .309 .379 ruby .276 .337 .219 .268 .279 .341 .219 .268 .248 .303 codebertscore-f1 .244 .298 .219 .268 .264 .324 .214 .262 .235 .288 codebertscore-f3 .281 .344 .243 .297 .313 .384 .261 .320 .275 .336 ice-score .330 .345 .313 .321 .294 .298 .315 .323 .313 .322 ref-ice-score .412 .438 .367 .383 .425 .446 .432 .455 .409 .431 with this implication, where the average score of the reference code only achieves 3.4 out of 4, suggesting that not all human practitioners consider the reference fully useful. ## discussion data contamination evaluations on recent closed-source llms have been criticized for the possibility of data contamination @xcite , where the model may have already seen the evaluation datasets during training, due to the opaque training details of these models. for instance, @xcite conducted an empirical study on a few closed-source llms, including gpt-3.5, and suggested that llms are the state-of-the-art evaluators of translation qual-ity, based on the evaluation of the wmt22 metric shared task @xcite . however, as most of the evaluated models were trained on data prior to 2022 foot_3 , it is highly likely that these models have been trained with some human-rated translation quality data. similarly, g-eval @xcite shows that gpt-3.5 and gpt-4 are the stateof-the-art evaluators of natural language generation (nlg) with the evaluation of three nlg datasets. however, as these human-annotated datasets were released before 2021, it is probable that they were included in the training data of gpt-3.5 and gpt-4. in contrast, our work is minimally impacted by data contamination, as we report the data release year in ## conclusion in this paper, we propose a novel evaluation metric based on large language models trained on both text and code, which can better capture the complex syntax and semantics of code intelligence tasks. our metric achieves superior correlations with subjective human preferences and objective functional correctness, both at the example and corpus levels, without reference and test suites. we conduct an extensive evaluation of four programming languages (java, python, c, c++, and javascript) and demonstrate the effectiveness of our proposed method on human-based usefulness and execution-based functional correctness. we have publicly released our evaluation metrics and datasets to encourage the development of more accurate and effective evaluation metrics for tasks involving source code.