Rethinking Code Refinement: Learning to Judge Code Efficiency (2024)

Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in understanding and generating codes. Due to these capabilities, many recent methods are proposed to automatically refine the codes with LLMs. However, we should rethink that the refined codes (from LLMs and even humans) are not always more efficient than their original versions. On the other hand, running two different versions of codes and comparing them every time is not ideal and time-consuming. Therefore, in this work, we propose a novel method based on the code language model that is trained to judge the efficiency between two different codes (generated across humans and machines) by either classifying the superior one or predicting the relative improvement. We validate our method on multiple programming languages with multiple refinement steps, demonstrating that the proposed method can effectively distinguish between more and less efficient versions of code.

## introduction large language models (llms) have shown significant success across a wide range of tasks, extending from natural language understanding to programming-related activities @xcite @xcite @xcite . specifically, thanks to their capabilities in understanding and generating codes, llms are able to allow developers to save time, reduce errors, and boost their productivity @xcite . for example, several recent studies have utilized llms to optimize and refine the existing code bases @xcite . also, more recent work has been proposed to iteratively refine the generated codes from llms by judging them with llms @xcite . another noteworthy approach involves using llms to check the functional correctness of the generated codes from llms @xcite . however, despite huge advancements made in the field of llms for code generation, the aforementioned studies assume that the codes generated and refined from llms are more efficient than their originals. however, as shown in figure 1 , our observations contradict this assumption, showing that llm-generated and -refined codes do not always perform better. to handle this issue, while one may calculate the efficiencies of codes both before after modifications by actually executing them, this process introduces unnecessary inefficiencies, and may be very costly and time-consuming. in this work, to overcome those challenges, we introduce a new task of judging the efficiency of the refined code over its original version. in addition, not only llms but also human coders may degrade the efficiency of codes during refinement; thus, our task of judging the efficiency between two codes involves all the possible pairs of code modification sources, including human-human, human-machine, and machine-machine. then, to address this new task, we propose a new model (based on the code lm) that is trained to compare efficiencies between two codes. specifically, given a code pair (one is original and the other is refined from it), the code lm is trained to classify which one is more efficient or predict how much the refined code is efficient. we experimentally validate the effectiveness of our efficiency judgement model on multiple code refinement scenarios with multiple programming languages. the results show that our judgement model can identify the more efficient code among two different versions, substantially outperforming baselines. our contributions are as follows: llms for code refinement beyond basic code generation, llms are widely used to refine the existing code bases. one of the early work in code refinement aims to detect and fix bugs in the codes by pre-training a transformer-based model on english and source code, and then fine-tuning it on commits (relevant to the part fixing bugs and improving performance) @xcite . in a similar vein, another work proposes to refine the code with a sequence-to-sequence model that is trained to transform the original code to its optimized version of the code @xcite . additionally, recent work showcases that llms are able to recursively improve their own generated codes, progressively enhancing their outputs @xcite . further, @xcite demonstrate that llms with sophisticated prompting strategies (to adapt llm for code optimization) can surpass human-level performance in code opti-mization tasks. however, despite these substantial achievements, prior studies have primarily focused on code enhancement with limited attention to the actual efficiency of the refined code. meanwhile, we focus on a different angle, proposing to judge the efficiency of the refined codes in advance (without executing them) based on our observation that not all the refined codes have better efficiency. ## method we first provide a general description of code refinement, and then introduce our approach. ## experiment we now describe experimental setups and results. we provide our code at https://github.com/goingdoer/judge_code_efficiency , for reproducibility. ## conclusion in this work, we pointed out that the codes refined by humans or machines are sometimes inferior than originals, and to tackle this, we introduced a novel approach to identify the more efficient code given a pair of codes before and after modifications. we validated our method on multiple code editing scenarios involving both humans and machines, showcasing its substantial efficacy despite its simplicity.