Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective (2024)

Abstract: Code generation aims to understand the problem description and generate corresponding code snippets, where existing works generally decompose such complex tasks into intermediate steps by prompting strategies, such as Chain-of-Thought and its variants. While these studies have achieved some success, their effectiveness is highly dependent on the capabilities of advanced Large Language Models (LLMs) such as GPT-4, particularly in terms of API calls, which significantly limits their practical applicability. Consequently, how to enhance the code generation capabilities of small and medium-scale code LLMs without significantly increasing training costs is an appealing challenge. In this paper, we suggest that code comments are the natural logic pivot between natural language and code language and propose using comments to boost the code generation ability of code LLMs. Concretely, we propose MANGO (comMents As Natural loGic pivOts), including a comment contrastive training strategy and a corresponding logical comment decoding strategy. Experiments are performed on HumanEval and MBPP, utilizing StarCoder and WizardCoder as backbone models, and encompassing model parameter sizes between 3B and 7B. The results indicate that MANGO significantly improves the code pass rate based on the strong baselines. Meanwhile, the robustness of the logical comment decoding strategy is notably higher than the Chain-of-thoughts prompting.

## introduction recently, techniques on pre-trained code large language models (code llms) are rapidly @xcite and perform well on * work done when yijie was interning at pattern recognition center, wechat ai, tencent inc, china. to address the above issue, we propose the idea for the first time that code comments serve as the inherent logical pivot bridging natural language and programming language. generally, comments within the code are commonly integral to the code corpus. consequently, during the pre-training stage, training on code corpus endows the pre-trained code models with the respective capacities for understanding and generating code comments. as depicted in figure 1 , code comments can decompose the problem description using natural language, and each comment line establishes an alignment with the neighboring lines of code. therefore, we hypothesize that encouraging models to generate comments can easily and effectively bridge the code and complex problem descriptions. in order to test our hypothesis, we discuss the possible inserting position of intermediate steps and use crossentropy loss to evaluate the difficulty of various inserting style codes. according to the statistical results, we found that code in comment style is the easiest for various code models, which provides evidence for the hypothesis. based on the above hypothesis, we propose comments as natural logic pivots (mango) for code generation with problem description decomposition via comments. mango includes a logical comment decoding strategy and comment contrastive learning loss. specifically, in the training phase, we generate negative samples without comments using the code data with comments in opensource datasets to strengthen the model preference for code with comments. during the decoding stage, the logical comment decoding strategy is adopted to guide the model in explaining the code logic via inline comments. we conduct experiments on humaneval and mbpp test sets using three backbone models from 3b to 7b. the experimental result shows that mango improves the pass rate consistently, e.g., up to 7.52 pass@10 on humaneval for starcoder-7b and up to 4.17 pass@10 on mbpp for wizardcoder-7b. our ablation studies show that each component of the method positively contributes to the performance of models, and the method is robust on the hyper-parameter and the various logical comment prompting styles. furthermore, mango keeps consistent effectiveness in small-size models (e.g., with pass@10 improvements of up to 3.87 on 3b and 4.07 on 7b), while cot prompting can lead to severe decline, especially in 3b models. the error distribution and code feature data statistics are also provided for fine-grained analysis. in summary, our contributions are as follows foot_0 : ## preliminary analysis: token-level loss of comment related code styles braking difficult problems into easier intermediate steps benefits the code generation ability of llms in many existing works especially for api-based close-source models. however, most of the task decomposition methods can work only on the most advanced models like gpt-4, and their applicability to open-source models with small or middle sizes is limited. therefore, we hypothesize that lowering the method difficulty of decomposition methods can benefit lowering the requirement of the model ability, and using comment is one of the possible methods. we define the possible styles for code with comment explanations and quantify their difficulties to models in zero-shot settings. first, we utilize wizardcoder-3b and wizardcoder-7b as backbone models, and the open-source code generation instruction tuning dataset codem as an evaluation dataset. we filtered the data with comments, extracted the comment line, and transferred it to the chain of thoughts. then, given a task decomposition step is t i and the corresponding code block is c i (where 1 ≤ @xmath0, and n is the code block number), we construct three types of code, including the code with the preceding chain of thoughts (cot-pre, which can be presented as {t 1 , ..., t n , c 1 , ..., c n }), the following chain of thoughts (cot-post, which can be presented as {c 1 , ..., c n , t 1 , ..., t n }), and the inline comments (comment, which can be presented as {t 1 , c 1 , ..., t n , c n }). the transferred cot-style datasets contain equivalent information compared with the origin comment-style dataset. the token-level cross-entropy loss can reflect the degree to which the model fits the specific token with the given context. assuming the evaluated token is t i , and the given context is @xmath1, • • • , t 0 }, the token-level cross-entropy of the token t i is l(t i ) . the l(t i ) can be calculated as equation 1 . using the token-level cross-entropy loss, we calculate the mean loss of the three datasets in different size models, including 3b and 7b. as shown in figure 2 comments contain equivalent information with the lowest difficulty. we analyzed the output codes and found that the loss scores of natural language tokens are usually higher than those of programming language tokens, which is also mentioned in @xcite . however, when the intermediate steps of natural language are inserted in code lines as comments, their loss score decreases obviously. in ## mango we proposed a simple and effective method mango, including comment contrastive learning loss and logical comment prompt. ## conclusion in this paper, we first emphasize the importance of code comments in bridging natural language and code language, and then we hypothesize that more comments to explain code logic can enhance code generation performance. we propose a simple and effective method mango, which contains contrastive comment loss and logical comment prompts. the main experiments and ablation studies show that mango effectively augments the pass rate of code generated by the models. in further analysis, we also compared lcp and cot prompting strategies and found that lcp has a significantly better and stabler performance than cot, especially for smaller models. the comment contrastive training can also boost cot performance in certain conditions. in conclusion, our work introduces the comment pivot as a novel perspective, and guiding models to use comments can augment the code generation ability stably. in the future, we will further explore the application potential of comments in broader complex scenarios, such as tool usage and llm agents.