Pass-Tuning: Towards Structure-Aware Parameter-Efficient Tuning for Code Representation Learning (2023)

Abstract: Code pre-trained models (CodePTMs) have recently become the de-facto paradigm for various tasks in the domain of code intelligence. To achieve excellent performance, the widely used strategy is to fine-tune all the parameters of CodePTMs. However, as the model size increases along with the number of downstream tasks, this strategy becomes excessively expensive. There are also some prior works that utilize Parameter-Efficient Learning (PEL) methods for model tuning in natural language processing to mitigate similar problems, but applying them directly to CodePTMs fails to capture the inherent structural characteristics of codes. To address the problem, in this paper, we propose Pass-Tuning for structure-aware Parameter-Efficient code representation learning. Specifically, a plug-and-play graph neural network module that can learn from Abstract Syntax Tree (AST) is employed as a tunable prefix. On the one hand, Pass-Tuning can further exploit the structural information of source code. On the other hand, it could serve as a replacement for full fine-tuning. We evaluate our method on multiple tasks across eight programming languages, including code understanding and generation. These results demonstrate the effectiveness, robustness, and universality of our method.

## introduction pre-trained language models @xcite have significantly boosted a series of natural language processing (nlp) tasks. these models mainly adopt deep transformer architecture @xcite , which are pretrained on a large-scale unsupervised text corpus, and then fine-tuned on downstream tasks. when regarding a code snippet as a sequence of tokens, it innately lends itself to these transformer-based * corresponding author. models @xcite @xcite ) from nlp. under the assumption of "software naturalness" @xcite and inspired by the enormous success of pre-training, code pre-trained models (codeptms) have also been proposed and widely applied in the realm of code intelligence @xcite @xcite @xcite . similarly, a straightforward approach to adapting these large-scale codeptms to the downstream tasks (e.g., code summarization, code clone detection) is fine-tuning the whole model for each task. however, the number of downstream tasks is ever-increasing, and it is often necessary to fine-tune the whole model for a specific programming language in real-world applications @xcite . the number of combinations of tasks and languages leads to standard full fine-tuning solutions being prohibitively expensive and cumbersome. to reduce the cost of model tuning, researchers employ strategies like @xcite to condition language models with frozen parameters to perform specific downstream tasks. notwithstanding, we experimentally observe that current pel methods @xcite struggle to reach the same stunning effect in code representation learning as in the nlp domain foot_0 . motivated to address the cost-performance dilemma for tuning codeptms, we believe that a new parameter-efficient learning (pel) approach with versatility should be built beyond the nlp domain. in our work, we present a parameter-efficient tuning approach with graph attention network capturing code structural information from abstract syntax tree, namely pass-tuning, a versatile pel method with minor tuning cost and competitive performance, to serve as a lightweight alternative to fine-tuning. specifically, we first construct tunable graph attention network (gat) modules as tunable prefixes to capture the code structure information contained in abstract syntax trees (asts). then, we design a novel code retrieval strategy based on attention distribution and ast token distances for better initialization. in order to adapt our approach to more scenarios, we implement a pel framework for codeptms with different architectures. then, we evaluate pass-tuning on multiple tasks over eight different programming languages. experiments show that our method can provide competitive (and in some tasks even better) performance while only modifying less than 1% of the parameters than full fine-tuning, substantially reducing the computational overhead for per-task/per-language when tuning codeptm on downstream tasks. our main contributions can be summarized as follows: ## preliminaries this section introduces basic concepts and notations used in this paper. ## experiments in this section, we conduct extensive experiments to evaluate pass-tuning and compare it against full fine-tuning and strong baselines. ## conclusion in this work, we present a novel pass-tuning framework for adapting codeptms with different architectures to downstream tasks. to our best knowledge, we are the first to propose a lightweight alternative to full fine-tuning in the domain of code representation learning and significantly reduce the number of trainable parameters for a series of tasks. besides, we consider source codes' structure information and employ tunable prefixes with gal architecture. moreover, a distinct initialization strategy is designed for prefixes to exploit the characteristics of codes. extensive experiments have demonstrated the effectiveness of our method and pass-tuning can steadily outperform all pel baselines. in comparison with full fine-tuning, we only modify less than 1% of the parameters while achieving competitive results. further ablation studies indi-cate the robustness and rationality of our strategies. in our future work, we will explore more parameter-efficient learning approaches for code and move one step forward to further utilize code structure information for tuning codeptms.