Metric-Oriented Pretraining of Neural Source Code Summarisation Transformers to Enable more Secure Software Development (2024)

Abstract: Source code summaries give developers and maintainers vital information about source code methods. These summaries aid with the security of software systems as they can be used to improve developer and maintainer understanding of code, with the aim of reducing the number of bugs and vulnerabilities. However writing these summaries takes up the developers’ time and these summaries are often missing, incomplete, or outdated. Neural source code summarisation solves these issues by summarising source code automatically. Current solutions use Transformer neural networks to achieve this. We present CodeSumBART - a BART-base model for neural source code summarisation, pretrained on a dataset of Java source code methods and English method summaries. We present a new approach to training Transformers for neural source code summarisation by using epoch validation results to optimise the performance of the model. We found that in our approach, using larger n-gram precision BLEU metrics for epoch validation, such as BLEU-4, produces better performing models than other common NLG metrics.

## introduction software documentation, such as method summaries, aids developers and maintainers in understanding how a software system works. venigalla and chimalakonda (2021) report that "software documentation aids better project comprehension and plays a major role in improving the popularity of the repository and also in increasing contributions to the repository. software documentation is capable of aiding various phases of software development, and maintenance". @xcite note the importance of code comments for program comprehension for software maintenance. the use of method summaries and other forms of code comment in reviewing code is vital for understanding that code. this review process can be used to find bugs and potential vulnerabilities in a codebase before they affect users. the united kingdom's national cyber security centre recommends both peer review as well as documenting and commenting code clearly as part of their recommended actions for secure development (national cyber security centre, 2020). however, @xcite note that "secure code development requires cognitive effort, and under constraints of time and resources developers struggle to keep security at the top of their priority list", meaning that practices relating to secure development are often not a primary concern, even for security-conscious developers. neural source code summarisation (nscs) aims to reduce his cognitive load on developers by summarising source code methods without developer interaction, using neural network models. nscs models require extensive training on large datasets of source code and related summaries to produce outputs with often low similarity to humanwritten summaries. our training produces a model which produces better outputs while requiring no more training than other, similar-sized models. nscs has grown in recent years with the development of new task-specific models, many of which build on @xcite 's transformer architecture, such as neuralcodesum @xcite and codebert @xcite . when training transformer models for summarisation tasks, each epoch of training can be validated against a natural language generation (nlg) metric. nlg metrics are often calculated alongside a loss metric or loss function, which is used to optimise the model during epoch validation. our training method takes a different approach by removing the reliance on loss for validating a training epoch. as is usual in model training, we use cross entropy loss during each training step to adjust model weights, but we opt not to use this in our epoch validation for early stopping, or for checkpointing. validation with loss or nlg metrics al-lows for "checkpointing" where the improvement in outputs from each epoch of training can be compared to previous epochs and the training can be stopped early if the training is no longer improving. the use of early stopping and checkpointing prevents overfitting to a given dataset by ensuring the outputs remain generic. while loss is still used to generate model weights, our method only uses an nlg metric for validating each training epoch. we train a bart transformer model @xcite on a source code summarisation task using a variety of validation metrics. we present a method of optimising pretraining to provide better results by monitoring the validation metric used, and checkpointing the best performing epoch. when an epoch fails to improve, the model weights are reverted to the best performing epoch, and the training continues. after 5 training epochs have failed to improve and a minimum of 20 training epochs have taken place, training stops. we discuss this in detail in section 3. ## dataset in order to train, validate, and evaluate the models, we use the filtered version of @xcite 's funcom dataset of java source code method -english language summary pairs, as done in previous works by @xcite and @xcite . we clean the dataset following @xcite 's approach, using their java implementation of the dataset cleaning tool foot_0 . @xcite 's method cleans the dataset using the matched pairs of java source code and javadoc comments. the cleaning method uses javaparser @xcite to select only compilable java code and remove inline code comments. it then finds the method summaries from the javadoc by extracting the first line of text with more than eight characters. we then follow @xcite 's steps: remove html and special characters (characters which are not alphanumeric, full-stops, apostrophes, or white space) from the summary and lowercase it. repeated method-summary pairs are then removed from the dataset, which is trimmed from 1.2 million pairs to roughly 500,000 pairs and split randomly into 80% training, 10% validation, and 10% evaluation datasets. this is the same split used by @xcite , @xcite , and @xcite . our dataset contains 499,997 method-summary pairs from multiple projects, split randomly into training, validation, and evaluation, as per ## research methodology we began by selecting the metrics we would use for validating models during training and evaluating models. the metrics chosen are as shown in in addition to these common summarisation metrics, we measure frugalscore and bertscore, which utilise llms to compare if the meaning of a machine-generated text matches the meaning of a human-written one, rather than whether the language used matches. llm-based metrics achieve this by capturing contextual embeddings. the forward step of the model training remains un-metric bleu-1 & 4 & smoothed bleu-4 meteor frugalscore bertscore our training and validation process is shown in figure 1 . our training dataset split of 399,999 method-summary pairs is used in the training step. as we validate our model, we use a validation split of 49,999 pairs. we use this data to calculate an nlg metric, then compare the average metric result to previous validation steps. if the model has improved in the last 5 epochs (early-stopping mechanism, x in figure 1 ) and the model produced the highest average metric score this epoch, these model weights are saved as a checkpoint, and the next epoch of training begins unless the maximum number of training epochs (n in figure 1 ) has been reached. if the model has shown improvement in the past 5 epochs, but has not improved in this training epoch, the model weights are reverted to the best scoring checkpoint. when this takes place, a small amount of noise is added to the weights in order to better prevent overfitting to the dataset and to prevent the model from generating the same model weights as the previous attempt. for this purpose, we added gaussian noise multiplied by 0.001 to each of the model weights individually. if the model has not improved in the last 5 epochs, the early stopping mechanism is called. when the early stopping mechanism is called, or the maximum number of training epochs has been reached, we evaluate the model against all of the metrics, using the evaluation dataset split of 49,999 methodsummary pairs. to ensure reliable results, we set a minimum of 20 training epochs. the results of our evaluation can be found in tables 4 , 5 , and 6. ## related work in 2021, @xcite analysed ten years of research into developer behaviour regarding secure coding practices, with regards to developer psychology, discovering barriers developers face to secure coding. later, @xcite identify an extensive list of security risks in practice, including a lack of secure development or coding. similarly, @xcite conducted a study of security practices in agile development. they found that while security is implemented in various ways in agile environments, models such as ssdlc for ensuring secure development are rarely implemented in their entirety. they also note a positive correlation between increased agility and increased security engineering practices. the transformer neural network model was introduced by @xcite as a general-purpose neural network. since then, the transformer has become a ubiquitous model for many nlp tasks. much work has been done to advance the transformer model; bart @xcite uses an architecture which combines both bidirectional and auto-regressive transformers to build a model. @xcite introduced t5, a simple transformer model, which treats all tasks as text-to-text problems, using transfer learning. optimising model training by optimising a model's parameters with respect to evaluation metrics is a concept initially developed by @xcite in the form of minimum risk training (mrt). mrt aims to optimise model parameters by minimising loss in terms of evaluation metrics. @xcite present an alternative algorithm, reward augmented maximum likelihood (rml). rml builds on maximum likelihood estimation, adding a step where log-likelihood is optimised on rewards for possible outputs. recent works have applied the transformer model architecture to nscs. codebert @xcite and neuralcodesum @xcite use transformer-based models to summarise source code, with codebert being a bidirectional transformer model. @xcite compare these two transformer models, as well as code2seq @xcite on the funcom dataset @xcite . @xcite establishes a method of cleaning funcom to allow for better training and evaluation of a neuralcodesum model, as well as introducing the use of an llm-based metric for evaluating nscs. recent work by @xcite focuses on altering the training process to produce better models for nscs tasks by using label smoothing. @xcite propose an alternative improved training approach for models for nscs tasks by using "meta-learning" to transform the training process into a few-shot deep learning task. @xcite propose a model, stunt, built on t5 small , for nscs tasks. stunt's training relies on a comment classification model, saloon, for generating training data as it is trained on snippets of code and related summaries found in code comments, not methods and method summaries. @xcite 's asm2seq model is designed to generate natural language summaries of x86 and amd64 assembly code for the purpose of aiding in vulnerability analysis. @xcite take a human approach to evaluating source code summarisation. @xcite found that "data suggests that participants did not see a clear difference in quality between human-written and machine generated comments"; finding developers' ratings to be an unreliable predictor of how much a summary helps them -and that developer intuition may be poor at assessing the relevancy of information. large language models have increasingly been used to generate metrics for nlg tasks. bertscore @xcite and moverscore @xcite being two examples of these metrics. these are large models, with a sizeable environmental impact when implemented at large scale. kamal eddine et al. (2022)'s frugalscore seeks to solve this by reducing the number of parameters used while retaining accuracy. frugalscore learns from the internal mapping of llms to produce a smaller language model with similar accuracy. ## conclusion we present codesumbart, an improved transformer model for automatic source code summarisation. our model uses a new training method to achieve a high degree of accuracy by validating the results of each training epoch against an nlg metric and using that validation performance to revert model weights from under-performing training epochs to those from the best-performing epoch. our findings show that our training provides an improved method of training transformer models for automatic source code summarisation. codesumbart outperforms state-of-the-art models in evaluation across several metrics and produces outputs comparable to human-written summaries to within a high degree of accuracy in two java source code summarisation tasks. this model can be applied to java source code methods to aid in the secure development process by reducing the cognitive load on developers. the model and training method we have created could be used to enable more secure software development through integration into developer tools to summarise new source code methods as they are written, and summarise legacy code methods for easier maintenance. following this work, we intend to continue to investigate the role that nscs models can play in cybersecurity, focussing on the potential application of nscs on bug and vulnerability patch data, using human evaluation alongside nlg metrics.