C ode P rompt: Task-Agnostic Prefix Tuning for Program and Language Generation (2023)

Abstract: In order to solve the inefficient parameter update and storage issues of fine-tuning in Natural Language Generation (NLG) tasks, prompt-tuning methods have emerged as lightweight alternatives. Furthermore, efforts to reduce the gap between pre-training and fine-tuning have shown successful results in low-resource settings. As large Pre-trained Language Models (PLMs) for Program and Language Generation (PLG) tasks are constantly being developed, prompt tuning methods are necessary for the tasks. However, due to the gap between pre-training and fine-tuning different from PLMs for natural language, a prompt tuning method that reflects the traits of PLM for program language is needed. In this paper, we propose a Task-Agnostic prompt tuning method for the PLG tasks, CodePrompt, that combines Input-Dependent Prompt Template (to bridge the gap between pre-training and fine-tuning of PLMs for program and language) and Corpus-Specific Prefix Tuning (to update the parameters of PLMs for program and language efficiently).Also, we propose a method to provide richer prefix word information for limited prefix lengths. We prove that our method is effective in three PLG tasks, not only in the full-data setting but also in the low-resource setting and cross-domain setting.

## introduction as the software engineering field continues to grow, the use of ai to increase the efficiency of developers through code intelligence is becoming increasingly important. in particular, program and language generation (plg) tasks, such as code summarization, code generation, and code translation, are essential for developers to maximize their productivity. code summarization allows developers to quickly understand the structure and purpose of a code, code generation assists by automatically generating code given a natural language description, and code translation facilitates the translation of code from one programming language to another, such as from java to c#, and vice versa. the recent success of pre-trained language models (plms) for the plg tasks, such as code-bert @xcite , plbart @xcite , and codet5 @xcite , has been attributed to their utilization of large-scale code and text corpora. the "pre-training then finetuning" approach has been widely used to derive program language representations by selfsupervised training on large-scale unlabeled data, which can then be transferred to multiple downstream tasks with limited data annotation. these approaches have proven to be successful on coderelated downstream tasks. however, fine-tuning large pre-trained models can be expensive and timeconsuming in terms of both updating and storing all parameters. furthermore, there is a discrepancy between pre-training and fine-tuning in the viewpoint of the inputs and the training objectives @xcite . it makes the model difficult to fully utilize the knowledge of pre-trained models, resulting in suboptimal performance on downstream tasks @xcite @xcite . in order to address the issues of fine-tuning, prompt tuning approaches have recently been proposed in natural language generation (nlg) tasks. to reduce the gap between pre-training and fine-tuning, @xcite proposed a prompt tuning method that combined manually crafted templates with the input. however, finding the optimal manual prompt template for each natural language task is arduous and laborious. furthermore, due to the updating of all parameters of the language model, it also requires updating full model parameters for each task, similar to fine-tuning. it can also easily lead to sub-optimal language model parameters in low-resource settings. li and liang (2021) proposed a lightweight method, prefix tuning, to freeze the language model "in the mid-1970s punk rock was born in a dank little new york nightclub called cbgb\'s. it all started when rockers like television, the ramones and patti smith launched a frontal assault on the monolith of corporate rock \'n roll. now another artistic revolt, remodernism, is about to widen its offensive from the birthplace of punk." on @xcite stedelijk museum and the university of amsterdam staged a talk on remodernism by daniel birnbaum, contributing editor of artforum, and alison gingeras, assistant curator, guggenheim museum. the summary is: in august 2006, an online group called "the remodernists of deviantart" was founded by clay martin. the group is composed of artists who are active on the website deviantart.com. in 2006, artist matt bray said, (a) an example of wikipedia def change_return_type(f): # converts the returned value of wrapped function to the # type of the first arg or to the type specified by a kwarg key # return_type's value. @wraps(f) def wrapper(*args, **kwargs): if kwargs.has_key('return_type'): @xmath0.pop('return_type') return return_type(f(*args, **kwargs)) elif len(args) > 0: @xmath1, **kwargs)) else: return f(*args, **kwargs) return wrapper parameters and instead optimize a sequence of continuous task-specific vectors (prefix). this method has shown performance comparable to fine-tuning in the full data setting while updating much fewer training parameters of a large pre-trained model. also, even in the low-resource setting, this method has been proven to be effective by prefix initialization with words specific to the task. however, those prompt tuning approaches for most nlg tasks are difficult to apply directly to the plg tasks. the pre-training of plms for natural language involves the use of large amounts of text data consisting of a series of sentences. this dataset contains task-specific natural language instructions (templates). as shown in figure 1a , task-specific natural language instructions, such as "summarize _", "tl;dr _", and "the summary is _", appear in data for pre-training. these templates can bridge the gap between pre-training and fine-tuning. however, datasets of plms for program and language hardly contain such task-specific textual instructions (templates). plms for program and language are usually pre-trained with unimodal data (code-only) or bimodal data (code-comment). input is either code or its corresponding comment, but there are very few task-specific natural language instructions, shown in figure 1b . due to the lack of task-specific instructions in the pre-training stage, it is hard to manually select task-dependent prompt templates for the plg tasks during the fine-tuning stage. prefix tuning in the nlg tasks improved performance by initializing prefix embedding from task-specific words, especially in low-resource settings. however, for plg tasks, we can hardly adopt such initialization approaches because there are very few task-specific words in data for plg tasks, as mentioned. also, unlike nlg tasks, plg tasks are two cross-modal generation tasks. it is not appropriate to initialize encoder and decoder prefixes with the same words in the same language. prefix embeddings of encoder and decoder need to be initialized with different words of their corresponding language. therefore, we propose a task-agnostic prompt tuning method, codeprompt, applicable to any plg tasks. our method consists of three components: input-dependent prompt template, corpusspecific prefix tuning, and multi-word prefix initialization. first, we propose the input-dependent prompt template by combining the template with input to bridge the gap between pre-training and fine-tuning in plms for program and language. input-dependent prompt template contains unified backbone words and input-specific words, regardless of the task. second, we propose the corpusspecific prefix tuning to reduce the number of parameters for update considering the traits of two cross-modal tasks. they can effectively transfer the task and corpus specific information in cross-modal tasks, especially in low-resource settings and zeroshot settings. third, we propose the multi-word prefix initialization to provide richer information to prefix embeddings while maintaining the number of parameters within the limited prefix length. our codeprompt shows great performances on three plg tasks in full data, low-resource, and crossdomain settings. ## codeprompt in this section, we explain our prompt tuning method, codeprompt, in detail. our prompt tuning method aims to consider the traits of plms for program and language. figure 2 shows the architecture of our method, which is on the basis of the codet5 framework, including input-dependent prompt template, corpus-specific prefix tuning, and multi-word prefix initialization. ## conclusion in this work, we proposed codeprompt, a task-agnostic prompt tuning method for program and language generation tasks. our codeprompt combined input-dependent prompt template to bridge the gap between pre-training and fine-tuning of plms for program and language, and corpusspecific prefix tuning to efficiently update the parameters of plms. additionally, we proposed multi-word prefix initialization method to provide more rich prefix word information for limited pre-fix lengths. we demonstrated that our method is effective in three plg tasks, both in full-data and low-resource settings, as well as in cross-domain settings.