Code Needs Comments: Enhancing Code LLM s with Comment Augmentation (2024)

Abstract: The programming skill is one crucial ability for Large Language Models (LLMs), necessitating a deep understanding of programming languages (PLs) and their correlation with natural languages (NLs). We examine the impact of pre-training data on code-focused LLMs’ performance by assessing the comment density as a measure of PL-NL alignment. Given the scarcity of code-comment aligned data in pre-training corpora, we introduce a novel data augmentation method that generates comments for existing code, coupled with a data filtering strategy that filters out code data poorly correlated with natural language. We conducted experiments on three code-focused LLMs and observed consistent improvements in performance on two widely-used programming skill benchmarks. Notably, the model trained on the augmented data outperformed both the model used for generating comments and the model further trained on the data without augmentation.

## introduction the development of large language models (llms) has made remarkable strides across various domains, including the field of code understanding and generation. works such as codegen @xcite , @xcite , and code llama @xcite have achieved significant breakthroughs in the task of natural language to code (nl2code). moreover, aligning natural language descriptions with their corresponding execution code to expand code-related training corpus to further enhance the model's coding capabilities has become a research focus for scholars @xcite @xcite @xcite . code llama (rozière et al., * equal contribution. † corresponding author language #chars of comment #chars comment density c# 5.4b 30.8b 0.1764 c++ 6.6b 38.0b 0.1753 go 3.0b 19.6b 0.1553 java 12b 66.8b 0.1917 javascript 6.3b 46.9b 0.1352 php 5.1b 42.3b 0.1207 python 9.6b 44.1b 0.2187 ruby 0.9b 5.18b 0.1821 rust 1.1b 6.44b 0.1641 typescript 2.4b 20.1b 0.1207 average 5.3b 32.0b 0.1670 considering that the alignment between natural language and code has not yet been relatively explored, comments serve as a representative and crucial bridge between the two. therefore, the primary objective of this work is to explore the significance of comments. an intuitive supposition posits that an augmentation in training corpus that aligns code and natural language (comments) will invariably enhance the model's performance. to quantify this alignment, we initially delineate "comment density" as the ratio of the number of non-white characters in comments to the total number of non-white characters and then examine how different levels of comment density impact downstream tasks. as shown in to ensure that the code remains unchanged during llms generation and accelerate the generation process, we propose a constrained generation approach that generates data on a line-by-line ba-sis, thereby circumventing the procedure of llms deleting, modifying the original code or producing new code. considering the need to exercise caution in trusting the comments added by the model, we introduce a discriminator in this study to filter out extreme cases. the discriminator evaluates the generated comments and filters out samples that exhibit significant differences from the original code. in our experiments, we observe that utilizing llms for comments generation not only enhances the capabilities of the base model but also facilitates self-augmentation. the overall framework of this work is depicted in figure 1 we highlight our contributions as follows: ## method indeed, generating comments for existing code by using llms is not a simple task for us with two principal challenges. firstly, llms often struggle to effectively follow the "add comments" instruction, resulting in code loss or insufficient comment additions, especially for longer code files. secondly, generating comments for large-scale pretraining code data can be computationally expensive, leading to significant training costs for the entire model. appendix a is a bad case where llms fail to follow the instruction of "add comments". ## experiments we initially lay the foundation with empirical evidence on the llama 2 model @xcite , illustrating that the fortification of alignment between code and natural language-particularly through the amplification of comment density-profoundly influences downstream tasks. subsequently, we apply our proposed methodology to the code llama model @xcite , underscoring its capacity not merely to bolster weak baselines such as llama 2, but also to achieve self-augmentation on models like code llama, distinguished by their exceptional performance in code generation tasks. moreover, we have substantiated through the internlm2 @xcite which is the most recent state-of-the-art llm in the field. that the pl-nl alignment data, generated by codellama, retains its efficacy for other models. all models were validated on the hu-figure 3 : illustration of the constrained generation algorithm. during the generation process, the code will be directly copied into the output until it encounters the marker indicating the beginning of a comment (#, "' or """ for python). the commented portion is generated by the code comment generator until the end of the comment (\n, "' or """, correspondingly). maneval @xcite and mbpp @xcite datasets. ## conclusion in this paper, we propose a novel method of code data augmentation that generates comments for existing code. we validate its effectiveness on three different llms. this signifies a novel paradigm shift towards self-augmentation for code llms, thereby illuminating the latent potential for llms to self-evolve and enhance.