Exploring Data Augmentation for Code Generation Tasks (2023)

Abstract: Advances in natural language processing, such as transfer learning from pre-trained language models, have impacted how models are trained for programming language tasks too. Previous research primarily explored code pre-training and expanded it through multi-modality and multi-tasking, yet the data for downstream tasks remain modest in size. Focusing on data utilization for downstream tasks, we propose and adapt augmentation methods that yield consistent improvements in code translation and summarization by up to 6.9% and 7.5% respectively. Further analysis suggests that our methods work orthogonally and show benefits in output code style and numeric consistency. We also discuss test data imperfections.

## introduction recent years have seen the rapid development of pre-trained models (plms) to enable knowledge transfer from generic texts to specific downstream tasks @xcite . plms have been applied to the programming language domain as well, following the same paradigm of (continuing) training plms on code and text data, and then fine-tuning them for specific tasks @xcite . plms are often adapted to programming languages by including code-specific modalities as part of the input like serialized syntax trees and data flows @xcite @xcite . such works have outperformed rule-based tools in various tasks, e.g. the codexglue benchmark @xcite . despite the abundance of raw code available for pre-training, code data that meet downstream needs stay modest in size. this is due to the fact that, unlike texts, code datasets cannot be easily curated by people without programming knowledge. for example, code translation data in codexglue is sized at 10k, which is orders of magnitude smaller than their natural language counterparts that often include millions of instances @xcite . we are therefore motivated to enrich data in the fine-tuning phase of code plms, using automatic data augmentation (da) methods like backtranslation, monolingual, multilingual, and numeric augmentation. we extensively experiment on code translation, where a programming language is converted to another, and summarization, where a textual description is produced from a code block. even with limited resources, we can lift performance by 6.9% for translation and 7.5% for summarization compared to baselines. through manual inspection and extra evaluation measures, we demonstrate that our methods lead to desirable enhancements special to code, namely better output code style and number correctness. ## analysis upon inspecting the translation test outputs, we find that our data-augmented model is better exposed to the target java language: it has learned the java programming conventions instead of following the input code style. we present test instances focused on element retrieval methods, by listing sources, references, and outputs from the code-bert baseline and our bt-augmented model in we should note that in the translation test set a small proportion of code pairs seem to be divergent, which can lead to an inaccurate estimate of translation performance. we record a few examples of these imperfections in appendix b.1, but leave indepth investigation and refinement for future work. @xcite , as well as string manipulation (e.g. token noising, swapping, deletion) can be useful. nonetheless, these methods are not task-specific, meaning they could be more appropriate for the generic code pre-training stage. ## conclusion we adapt several data augmentation techniques to programming language translation and summarization. our investigation includes data synthesis, knowledge sharing via multilinguality, and numeric-aware techniques. enhanced performance is observed in experiments conducted on a variety of pre-trained code language models, and our analysis demonstrates that these methods can benefit output code style and numeric correctness.